We Have Reached the Clever Bit
The transformer does not read a sentence the way a human does. It repeatedly transforms numerical representations of the tokens.
At the end of the previous page, every token position had a numerical representation, with position handled according to the model's architecture.
Those representations now enter a stack of transformer blocks.
It is a repeated architecture containing several mathematical operations. Two of the most important are self-attention and a feed-forward / MLP network.
The Shape of One Transformer Block
Exact details vary among model families, but a modern decoder-style language-model block can be understood at a high level like this:
The exact ordering and type of normalization, attention implementation, MLP design, and other details are architecture-dependent.
Self-Attention: Tokens Can Use Information From Other Tokens
Suppose our prompt contains:
When processing a token representation, useful information may come from other positions in the sequence. The representation associated with it, for example, may need information associated with earlier tokens.
Self-attention provides a mathematical mechanism for combining information from token positions based on learned relationships.
It is a set of numerical operations that calculate relationships among representations and use those relationships to produce new representations.
Q, K and V
Here are the famous three letters.
For each token representation, learned matrix operations produce three new vectors commonly called:
Query
A transformed representation used to determine which information at allowed positions is relevant to the current position.
Key
A transformed representation used for comparison with queries.
Value
The information representation that can be weighted and combined into the attention output.
The words query, key, and value are useful labels, but these are vectors—arrays of numbers—not literal questions, database keys, or pieces of text.
Where Do Q, K and V Come From?
They are calculated from the incoming token representations using learned weight matrices that are part of the trained model.
Q = X × WQ
K = X × WK
V = X × WV
X represents the incoming token representations. WQ, WK, and WV represent learned parameter matrices.
This is exactly why loading the trained weights earlier mattered: those learned numbers are now actively participating in computation.
Queries Meet Keys
The attention mechanism compares a query with keys at positions the model is permitted to attend to.
In standard scaled dot-product attention, a dot product produces a numerical compatibility score. The scores are scaled and then passed through a softmax operation.
The resulting weights determine how strongly the corresponding value vectors contribute to the attention output for that position.
The Causal Mask: No Peeking Ahead
For a normal autoregressive language model, a token position is prevented from attending to future positions that should not yet be available.
token 2 → may use tokens 1–2
token 3 → may use tokens 1–3
token 4 → may use tokens 1–4
...
This is commonly enforced by an attention mask before the softmax weighting is finalized.
It can use the context already supplied to it, but the attention calculation is constructed so it does not obtain future generated tokens that do not yet exist.
Values Are Weighted and Combined
After attention weights have been calculated, the corresponding value vectors are multiplied by those weights and combined.
output = (weight₁ × V₁) + (weight₂ × V₂) + ... + (weightₙ × Vₙ)
The result is a new vector containing a mixture of information drawn from allowed token positions.
So a token representation that entered attention largely as its current state leaves attention carrying newly combined contextual information.
Not One Attention Calculation: Multiple Heads
Transformers normally use multi-head attention or related variants.
Different attention heads operate with different learned projections. This allows the model to compute multiple patterns of relationships in parallel.
Head 1
One learned projection of queries, keys, and values.
Head 2
Another learned projection operating on the same sequence representations.
Head ...
Additional heads provide additional learned relationship computations.
The outputs are then combined and projected back into the representation space used by the rest of the block.
We should not casually label individual heads as “the grammar head,” “the fact head,” or “the emotion head.” Researchers can identify specialized behavior in some heads, but the learned computation is distributed and model-specific.
Residual Connections: Don't Throw Away the Incoming Signal
The attention result does not simply replace everything that entered the block.
A residual connection combines the sublayer's result with the representation that entered it. This helps information and gradients move through deep networks and allows each block to refine an existing representation rather than rebuild it from nothing.
Then Comes the Feed-Forward Network
Attention is only part of the transformer block.
Each token position is also processed through a learned feed-forward network, often called an MLP. It applies additional learned transformations and nonlinear operations to the representation at each position.
The exact MLP design varies among model architectures. Modern models may use gated variants rather than the simplest two-linear-layer textbook form.
What Has One Block Accomplished?
The token representations that leave the block are not the same representations that entered it.
They have been transformed using:
Learned Weights
Parameters acquired during training have participated directly in the calculations.
Other Token Positions
Attention has mixed information from allowed positions in the sequence.
Nonlinear Transformation
The MLP has further transformed each position's representation.
Residual Information
Earlier representations remain connected to later representations through residual paths.
And Then We Do It Again
A useful modern language model does not contain only one transformer block.
The updated representations from one block become the input to another block, and then another.
Across this stack, the representations are repeatedly transformed. Contextual relationships become encoded in increasingly processed numerical states.
Where Is All This Happening?
Back inside our physical machine.
These operations become large tensor computations executed primarily on the GPU or other accelerator. The model weights are read from fast accelerator-accessible memory; intermediate activations are created; matrix multiplications and other kernels run massively in parallel.
At the physical level we have transistors switching, memory being read and written, and enormous numbers of arithmetic operations. The remarkable behavior emerges from the architecture, the learned weights, the input, and the repeated transformations they produce together.
Leaving the Transformer Stack
Our prompt entered as token representations.
After passing through many transformer blocks, we now have deeply transformed, context-dependent numerical representations.
But there is still no English answer on the screen.
The model must turn its final representation into a decision about what token should come next.
That requires one more transformation—from the model's internal representation to scores across its vocabulary.
HAAW Checkpoint
- Established mechanism
- Self-attention derives query, key, and value representations using learned parameters.
- Established mechanism
- Attention scores determine weighted combinations of value representations from allowed token positions.
- Established mechanism
- Transformer blocks also contain learned feed-forward/MLP transformations, normalization, and residual paths.
- Important distinction
- Attention is a mathematical operation, not evidence of conscious focus or intention.
- Architecture-dependent
- Normalization order, attention variants, positional mechanisms, MLP designs, number of heads, and number of blocks vary among models.
- Interpretive caution
- Individual heads, dimensions, or activations should not automatically be assigned simple human-readable meanings.
- Still ahead
- Turning the final transformer representation into vocabulary scores and selecting the next token.