AI Home › How AI Actually Works › Inference — Running a Query
Part 5 of 7 — How AI Actually Works

Inference — Running a Query

You hit Enter. What happens next?

Training built the model. The model is frozen. Now a user sends a prompt. Inference is the process of running that frozen model against a live input to produce a response. It happens in milliseconds, billions of times a day, on infrastructure you never see. This page walks every step from the moment you hit Enter to the moment a response appears.

The Inference Pipeline

1
Prompt received — your text arrives at the API endpoint over HTTPS
2
Tokenization — prompt text converted to a sequence of token IDs
3
Context assembly — system prompt, conversation history, and your prompt assembled into the full context window
4
Forward pass — token sequence runs through all transformer layers; attention, feed-forward, repeat for each layer
5
Token sampling — output layer produces probability distribution; next token selected based on temperature setting
6
Token appended — selected token added to the context; process repeats from step 4
7
Stop condition met — model generates an end-of-sequence token, or max token limit reached
8
Detokenization — token IDs converted back to text and streamed to the client

Worked Example — “What is the capital of France?”

Abstract steps are useful. Concrete examples are better. Here is the same pipeline traced through a single real query — every step shown as it actually happens on the inference server.

#
What happens
For this query
1
Prompt received
Text arrives at API endpoint over HTTPS
The string What is the capital of France? arrives as an HTTP POST to the inference server. The connection is encrypted — the model provider never receives it in plain text over the wire.
2
Tokenization
Text converted to token IDs
The prompt becomes a sequence of integers — something like [1867, 374, 279, 6864, 315, 9822, 30] — 7 tokens.
What is a token? A token is the model's basic unit of text — not a word, but a chunk. Common words become one token (France = 1 token). Rarer words split into pieces (cap-i-tal might be 2). Numbers and punctuation are often their own tokens. The model never sees letters or words — only these integer IDs. One token is roughly 0.75 words in English.
3
Context assembly
System prompt + history + query assembled
If this is a fresh conversation the full context is: the system prompt (perhaps 30 tokens) plus the 7 query tokens — roughly 37 tokens total out of a possible context window of hundreds of thousands. A long conversation would include all prior turns, growing the context with each exchange.
4
Forward pass
All tokens run through transformer layers
All 37 context tokens pass through 80+ transformer layers. At each layer, self-attention runs — every token looks at every other token and weighs its relevance. The word capital attends strongly to France because capital of [country] is a geographic pattern seen billions of times in training data.
Why not money? “Capital” can mean a city, money, or an uppercase letter. The model does not look up a dictionary — it calculates probabilities based on context. Here, of France is a geographic construction. Self-attention connects capital to France (a country) rather than to any financial term. The pattern capital of [country name] overwhelmingly maps to a city in training data, so that meaning dominates. A different prompt — “What is the capital gains tax in France?” — would produce a completely different attention pattern and a completely different response.
5
Token sampling
Probability distribution produced; next token selected 📖 Glossary
The output layer scores every token in the vocabulary (~100,000 tokens). Paris scores highest by a wide margin — it has by far the greatest probability of being the correct next token given this context. At low temperature the model picks it directly. Token ID for Paris is selected.
6
Token appended
New token added; forward pass repeats
The context now contains the original 37 tokens plus Paris. The forward pass runs again to determine what comes next. Given What is the capital of France? Paris, the model now generates a natural continuation — likely a period or a fuller sentence depending on the system prompt's style instructions.
7
Stop condition met
End-of-sequence token generated
After generating Paris. the model produces an end-of-sequence token — a special token learned during training to signal that the response is complete. For a simple factual query the response is short; the stop condition fires quickly. A complex question would generate many more tokens before stopping.
8
Detokenization
Token IDs converted back to text and streamed
The token IDs are converted back to the string Paris. and sent to your browser as it is generated — you see the word appear on screen. The entire process from your Enter key to the response appearing took well under one second.

The Forward Pass — What the Model Actually Does

Step 4 is where the model does its work. The entire token sequence — everything in the context window — passes through the transformer layers one by one. Each layer applies self-attention (every token looking at every other token) and then a feed-forward transformation. A large model may have 80 or more of these layers stacked on top of each other.

The output of the final layer is a vector of numbers — one number per token in the vocabulary — representing the model's confidence that each token should come next. A softmax function converts those numbers into probabilities that sum to 1.0.

This entire process — for a 70B parameter model — involves trillions of floating-point multiplications. On GPU hardware it completes in milliseconds. On a CPU it would take seconds to minutes per token, which is why GPU inference matters.

The sysadmin analogy: Think of inference like processing a request through a deep pipeline of filters — each stage transforms the data and passes it forward. Except instead of 5 or 10 pipeline stages, there are 80, each involving millions of matrix multiplications. The output of the last stage is a ranked list of what should come next. The highest-ranked item (or a sampled one) gets appended and the pipeline runs again.

Autoregressive Generation — One Token at a Time 📖 Glossary

This is the detail that surprises most people: the model does not generate the entire response at once. It generates one token at a time, appending each new token to the context and running the full forward pass again for the next one.

A 500-token response requires 500 forward passes. Each pass processes the entire growing context. This is why longer responses take longer — not just because there is more output, but because each token requires a full pass through all the model's layers over an increasingly long context.

This sequential dependency — each token depends on all previous tokens — is called autoregressive generation. It is fundamental to how transformer-based language models work and why parallelizing generation across tokens is an active area of research.

Streaming — Why You See Words Appear One by One

When you use Claude or ChatGPT and watch words appear progressively, you are seeing inference in real time. Each token is sent to your browser as soon as it is generated — the API streams tokens over the connection rather than waiting for the complete response.

This is better user experience (you see progress immediately) and also reflects the actual computation — the model genuinely is producing one token at a time. Streaming is not a UI trick; it is the natural output of autoregressive generation.

Batching — Serving Many Users at Once 📖 Glossary

An inference server does not process one request at a time. It batches multiple requests together and runs them through the GPU simultaneously. This is how a relatively small number of GPUs can serve millions of users — many requests share the same forward pass.

Batching introduces a tradeoff: waiting to fill a batch increases latency for individual requests but improves throughput overall. Inference infrastructure tuning is largely about finding the right batch size and scheduling strategy for a given workload and latency target.

Training vs Inference — The Infrastructure Difference

Training Infrastructure

Inference Infrastructure

KV Cache — The Inference Optimization You Use Every Day 📖 Glossary

Recomputing attention for every token in the context on every forward pass would be enormously wasteful. The KV cache (key-value cache) stores the attention computations for tokens already processed, so only the new token requires full computation on each pass.

The KV cache lives in GPU VRAM. Longer contexts require larger caches. This is another reason context window size is limited by VRAM — not just the model weights, but the cache for every active conversation competes for the same memory.

When an inference server runs out of KV cache space, it either rejects new requests or evicts older conversations. Managing this tradeoff is a core challenge of production AI infrastructure.

Reference Material

← Part 4: The LLM Engine Part 6: The Interface Layer →