Inside the model — transformers, tokenization, and how text becomes prediction.
A Large Language Model is, at its core, a very sophisticated next-token predictor. Given a sequence of tokens — the words and punctuation in your prompt — it calculates a probability distribution over every possible next token and picks one. Then it does it again. And again. One token at a time until the response is complete.
That sounds simple. The sophistication is in how it calculates those probabilities. The model has billions of numerical parameters — weights — that were shaped by training to encode patterns, facts, reasoning, and language structure. Those weights are what make the difference between a model that says something useful and one that says something plausible but wrong.
Computers work with numbers. Before the model can process your prompt, the text must be converted to tokens — integers that represent chunks of text. This is tokenization.
Tokens are not words. They are subword units — common words become a single token, less common words get split into pieces. The sentence "How AI Actually Works" might tokenize like this:
A technical term like "tokenization" might split:
The model never sees text. It sees sequences of integers. The tokenizer converts your prompt to integers going in, and converts integers back to text coming out. Modern models like Claude use vocabularies of roughly 100,000 tokens.
Token count matters practically — it determines cost (APIs charge per token), context window limits (models can only process so many tokens at once), and response length. A rough rule: one token is approximately 0.75 words in English.
The architecture that powers every major language model today is the Transformer, introduced in the 2017 paper Attention Is All You Need. Before the Transformer, language models processed text sequentially — one word at a time, left to right. The Transformer processes the entire sequence in parallel and — crucially — learns which parts of the input to pay attention to when generating each part of the output.
That mechanism is called self-attention. When the model processes the word "it" in a sentence, self-attention lets it look back across the entire context to figure out what "it" refers to. This is what gives language models their ability to handle long, complex passages with coherent understanding.
Converts token IDs into vectors — lists of numbers that position each token in a high-dimensional space where similar meanings cluster together. "King" and "Queen" end up closer together than "King" and "database".
The mechanism that lets every token look at every other token in the context and decide how much to weight each one. Multiple attention "heads" run in parallel, each learning different relationships.
After attention, each token's representation passes through a feed-forward network — two linear transformations with an activation function between them. This is where much of the model's factual knowledge is encoded.
Takes the final representation and produces a probability score for every token in the vocabulary. The highest-probability token (or a sampled one, for variety) becomes the next output token.
When you hear that a model has 70 billion parameters, those parameters are the numerical weights in all the layers described above — the embedding vectors, the attention weight matrices, the feed-forward layer weights. Each is a floating-point number, typically stored in 16-bit precision.
70 billion parameters at 16-bit precision requires roughly 140GB of storage — which is why a 70B model does not fit in consumer GPU VRAM and why quantization (reducing precision to 4-bit or 8-bit) exists. Quantization trades some accuracy for the ability to run the model on smaller hardware.
Every model has a context window — the maximum number of tokens it can hold in memory during a single interaction. This includes your prompt, the conversation history, any documents you attached, and the response being generated.
Earlier models had context windows of 4,096 tokens (~3,000 words). Modern models like Claude handle hundreds of thousands of tokens. The context window is a hard limit — tokens beyond it are simply not seen by the model. This is why very long conversations can cause a model to "forget" what was said at the beginning.
The context window is held in GPU VRAM during inference. Larger context windows require more VRAM — which is one reason increasing context window size is an engineering challenge, not just a software setting.
The output layer produces a probability distribution, not a single answer. Temperature controls how that distribution is sampled. At temperature 0, the model always picks the highest probability token — deterministic, consistent, sometimes repetitive. At higher temperatures, lower-probability tokens get more chances — more creative, more varied, sometimes less accurate.
When you notice Claude being more or less "creative" depending on the task, temperature (and related settings) are part of why. Code generation benefits from low temperature — you want the most likely correct answer. Creative writing benefits from higher temperature — you want variety.