One Token Is Not an Answer
The model has selected its first output token. That token now becomes part of the context used to generate the next one.
Then another token is selected. Then another.
This repeated process is called autoregressive generation.
It is generated incrementally, one token after another.
Two Phases: Prefill and Decode
It helps to divide inference into two computational phases.
Prefill
The existing input sequence—the prompt and supplied context—is processed through the model. Attention state needed for later generation can be computed and cached.
Decode
The model generates new tokens sequentially. Each step uses the prior context plus the newly generated token state to determine another next token.
These phases use the same trained model but have different computational characteristics.
Prefill: Process the Prompt
During prefill, the prompt's token representations move through the transformer stack.
Because all prompt tokens are already known, the accelerator can process substantial portions of their computation in parallel, subject to the causal attention structure.
Why Not Recalculate Everything?
Imagine the prompt contains hundreds or thousands of tokens.
After the first output token is selected, the earlier prompt has not changed. Much of the attention-related information already calculated for those earlier positions would be identical if recomputed.
Recalculating all of it for every new token would waste time and accelerator work.
This is the job of the KV cache.
The KV Cache
Remember Q, K, and V from self-attention?
During generation, previously computed key and value representations for earlier token positions can be retained for reuse in later attention calculations.
Without Reuse
For every generated token, repeatedly recompute earlier key/value state that has already been calculated.
With KV Cache
Keep prior key/value state available and compute the new state required for the newly added token position.
prompt token 1 → K,V cached
prompt token 2 → K,V cached
prompt token 3 → K,V cached
...
generated token 1 → new K,V added
generated token 2 → new K,V added
generated token 3 → new K,V added
What Is Not Cached?
The name matters: KV cache.
It is not simply a stored copy of every computation in the transformer. New computation still has to happen for each generated token.
New Token Representation
The newly selected token must be represented and processed.
New Queries
Attention still needs query information for the current token position.
New K and V
Keys and values for the new position are computed and can then be added to the cache.
Layer Computation
The new token still passes through the transformer's layers, including attention and MLP operations.
One Decode Step
Now we can follow one complete turn of the generation loop.
The Cache Grows With the Sequence
As more tokens are generated, more key/value state is retained.
This improves computational reuse, but it consumes memory. The amount depends on factors including model architecture, number of layers, attention design, sequence length, numerical representation, batching, and serving implementation.
More context can mean more memory devoted to cached attention state and more attention work during decoding.
Where Does the KV Cache Live?
For GPU inference, KV cache data is commonly kept in GPU-accessible memory because attention operations need rapid access to it.
Depending on the serving architecture, hardware, memory pressure, and optimization strategy, implementations may manage or move cache data differently.
This means VRAM is doing more than holding model weights.
Why Generation Often Feels Sequential
During prefill, many prompt-token calculations can be performed in parallel.
During ordinary autoregressive decoding, however, the next generated token depends on the token selected immediately before it.
↓
token 2 can then be selected
↓
token 3 can then be selected
↓
token 4 ...
There is enormous parallel computation inside each model step, but the ordinary token-generation dependency itself is sequential.
Streaming: Why We Can Watch the Answer Appear
The serving system does not necessarily wait for the entire response before returning anything to the user.
As generated token IDs are decoded into text, the application can send output fragments back to the client.
The visual effect of an AI “typing” is therefore commonly the client displaying pieces of a response while generation continues on the server.
Meanwhile, the Machine Is Busy
While a sentence appears calmly on the screen, the accelerator may be executing a large amount of work for every token.
Memory Reads
Model weights and cached state must be accessed repeatedly.
Matrix / Tensor Operations
Attention projections, MLPs, output projections, and supporting operations execute across the accelerator.
Cache Updates
New attention key/value state is retained as the sequence grows.
Token Selection
Vocabulary scores are converted into another concrete token choice.
Then the Model Stops
Eventually a stopping condition is reached: perhaps an end-of-turn token is generated, a configured limit is reached, a stop condition fires, or the request is interrupted.
The serving system then finishes the generation request.
Request-specific working memory and cache resources can then be released or managed according to the serving system's design.
Our Entire Prompt-to-Response Journey
We Have Completed One Inference Journey
We started with a powered-off computer.
We booted Linux, started the AI runtime, loaded the model, accepted a prompt, tokenized it, created vectors, passed those representations through transformer blocks, calculated vocabulary scores, selected tokens, reused KV state, and streamed the resulting text back to the user.
For the first time, we can trace a complete path from hardware startup to finished AI response.
But we have deliberately treated several mechanisms as boxes along the way.
Now HAAW can begin opening those boxes.
HAAW Checkpoint
- Established mechanism
- Autoregressive language models generate output sequentially, one token after another.
- Established mechanism
- Inference is usefully divided into prompt prefill and iterative decoding phases.
- Established mechanism
- KV caching reuses previously computed attention key/value state during later decoding steps.
- Important distinction
- The KV cache is computational state, not a human-like memory or a readable transcript database.
- Physical consequence
- KV cache, weights, activations, and working buffers all contribute to accelerator-memory demand.
- Important distinction
- Autoregressive generation is sequential between tokens even though enormous parallel computation occurs inside each token step.
- Journey complete
- We now have a continuous high-level mechanism from machine startup through a completed generated response.