The Transformer Has Finished This Pass
After many layers of attention and learned transformation, the model has produced context-dependent numerical representations.
For next-token generation, the representation at the relevant final sequence position is used to determine scores for possible next tokens.
The model still does not have an English sentence waiting somewhere inside it. It must choose the response one token at a time.
The model predicts one next token, appends that token to the sequence, and then continues the process to obtain another.
1. From Hidden State to Vocabulary Scores
The final internal representation is transformed into one score for each token in the model's vocabulary.
These raw scores are usually called logits.
If a model has a very large vocabulary, this produces a very large vector of scores—one candidate score for each possible next token.
What Is a Logit?
A logit is a raw numerical score produced before normalization into probabilities.
token_A → 8.2
token_B → 6.9
token_C → 5.1
token_D → 1.4
... thousands of other vocabulary tokens ...
A larger logit generally means the model currently assigns greater relative support to that token than to tokens with smaller logits.
But logits are not themselves probabilities. They do not have to add up to 1 or 100%.
2. Softmax Can Turn Scores Into a Distribution
A softmax operation converts a set of logits into normalized probabilities.
The exact values below are invented only to illustrate the idea:
They represent the model's next-token distribution given the sequence and context supplied up to this point.
3. Highest Probability Does Not Always Mean Selected
If the system always selected the token with the highest score, generation would be deterministic under otherwise identical conditions. This is often called greedy decoding.
But many deployments use a sampling strategy instead.
Greedy
Select the highest-scoring next token.
Temperature
Adjusts the shape of the token distribution before sampling. Lower values generally concentrate probability more strongly; higher values flatten it.
Top-k
Restricts sampling to a chosen number of the highest-scoring candidate tokens.
Top-p
Restricts sampling to a set of high-probability tokens whose cumulative probability reaches a chosen threshold.
Other decoding controls and implementation-specific methods also exist.
So Is the Choice Random?
Sometimes sampling includes a pseudorandom draw from a filtered or adjusted probability distribution. That does not mean every vocabulary token is equally likely.
A token assigned much greater probability remains much more likely to be selected than a low-probability token, unless the decoding procedure changes or excludes candidates.
With deterministic decoding—or with controlled random seeds and deterministic implementations—the same input may produce the same selection. With stochastic sampling, repeated runs can diverge.
4. One Token Is Selected
Eventually the decoding procedure produces one token ID.
That is the first concrete piece of newly generated model output.
It may represent a complete word, part of a word, punctuation, whitespace-related material, or another tokenizer vocabulary unit.
5. The Token Becomes Text
The tokenizer process can operate in the opposite direction as well. The selected token ID corresponds to token data that can be decoded toward human-readable text.
selected token ID → tokenizer decode → visible text fragment
Applications may stream decoded text to the user as tokens are generated, or buffer some output before displaying it.
But the Answer Is Not Finished
One generated token is now appended to the existing sequence.
The model now needs another token.
Does the Whole Prompt Go Through Everything Again?
Conceptually, each next-token prediction depends on the sequence that came before it.
But an efficient transformer inference system does not necessarily recompute every earlier attention quantity from scratch for every generated token.
During autoregressive generation, transformer systems commonly retain previously computed attention keys and values in a structure called the KV cache.
It stores computational state that allows later decoding steps to reuse prior key/value representations instead of recalculating all of them.
We will give the KV cache its own treatment when we follow repeated generation in detail.
When Does Generation Stop?
The cycle continues until a stopping condition is reached. Depending on the model and serving system, this may include:
End Token
The model generates a designated end-of-sequence or end-of-turn token.
Length Limit
The serving system reaches its configured maximum output or context constraint.
Stop Sequence
An application-defined token or text pattern tells generation to stop.
External Control
The application, user, tool framework, or serving environment terminates generation.
What Does “Predict the Next Token” Really Mean?
This phrase is accurate, but it can sound much simpler than the mechanism we have now followed.
Before one next-token distribution appears, the system has already:
The phrase next-token prediction describes the output objective. It does not mean the machinery producing that prediction is simple.
One Token Has Escaped the Machine
For the first time in our HAAW journey, the AI has produced something new.
One token.
Then that token becomes part of the context used to produce another token, and another, and another.
A response emerges incrementally from repeated inference.
Our next job is to follow that loop closely—and see why generation does not require the machine to start the entire computation from zero every time.
HAAW Checkpoint
- Established mechanism
- The final transformer representation is projected into vocabulary-wide scores called logits.
- Established mechanism
- Softmax can normalize logits into a probability distribution over candidate next tokens.
- Important distinction
- The highest-probability token is not necessarily selected when stochastic decoding is used.
- System boundary
- The neural model produces token scores; the generation procedure applies decoding rules and selects the token.
- Established mechanism
- The selected token becomes part of the sequence used for subsequent autoregressive generation.
- Efficiency mechanism
- KV caching allows prior attention key/value state to be reused during later decoding steps.
- Still ahead
- The repeated generation loop: KV cache, one-token-at-a-time decoding, streaming, and completion of the response.