The Transformer — Attention Is All You Need

Every LLM covered elsewhere in this module — its tokenizer, its neural network structure, its capabilities — sits on top of one architectural decision made in a single 2017 paper: Attention Is All You Need (Vaswani et al.). Before that paper, the leading approach to language modeling was fundamentally sequential. After it, the leading approach was fundamentally parallel. This page covers what problem that shift solved, how the core mechanism works, how the pieces fit together, and why the paper's confident title turned out to be correct.

1. The Problem It Solved

Before the Transformer, the dominant architectures for language tasks were Recurrent Neural Networks (RNNs) and their more capable variant, LSTMs (Long Short-Term Memory networks). Both processed text the way a person reads aloud one word at a time: sequentially, left to right, carrying a running "memory" of everything seen so far into the next step.

No parallelization during training: Because step N depended on the output of step N-1, an RNN couldn't process token 50 until it had finished processing tokens 1 through 49. Training was forced to run in serial lockstep, which meant it couldn't take advantage of GPUs' real strength — doing thousands of independent calculations at once.
Long-range relationships degraded: Information about token 1 had to survive being passed through 49 intermediate steps to still be usable at token 50. In practice, that signal faded — the "vanishing gradient" problem — so RNNs were weak at connecting ideas that were far apart in a long passage.
The compute didn't scale: Even with more data and bigger models, the sequential bottleneck capped how much practical benefit you could get from throwing more hardware at the problem.

Example: In the sentence "The trophy didn't fit in the suitcase because it was too big," an RNN has to carry the idea of "trophy" and "suitcase" all the way through six intervening words before it can even attempt to resolve what "it" refers to — and by that point, that signal has already started to fade.

The Transformer's answer was to remove the sequential dependency entirely. Instead of passing a single running memory step by step, every token looks directly at every other token in the sequence, all at once. That single change is what unlocked both massively parallel training and a much stronger grip on long-range relationships.

2. The Attention Mechanism

Self-attention is the mechanism that replaced sequential memory. For every token in a sequence, the model computes a weighted relationship to every other token — asking, in effect, "how relevant is each other word to understanding this word, right now?" Those weights aren't fixed; they're learned during training and recalculated fresh for every sentence the model processes.

Mechanically, each token is projected into three vectors:

Query — what this token is "looking for" from the rest of the sequence.
Key — what this token "offers" to other tokens searching for relevant context.
Value — the actual content this token contributes once it's been deemed relevant.

A token's Query is compared against every other token's Key to produce a relevance score; those scores are converted into weights (via softmax) and used to blend the Values into a new, context-aware representation of that token. Every token in the sequence does this simultaneously — which is exactly the parallelism the RNN couldn't offer.

Worked example: "The animal didn't cross the street because it was too tired." What does "it" refer to — the animal, or the street?

A human resolves this instantly using world knowledge (streets don't get tired). Self-attention resolves it structurally: when the model computes the representation for the token "it," its Query vector produces a high relevance score against the Key for "animal" and a low score against the Key for "street." The resulting blended representation of "it" ends up weighted heavily toward "animal" — the model has, in effect, linked the pronoun to its referent without needing to track it step by step through the intervening words.

Change one word — "The trophy didn't fit in the suitcase because it was too big" — and the attention pattern shifts: now "it" weights toward "trophy" instead of "suitcase." Same sentence structure, same position of the word "it" — but a different attention pattern because the content changed. This is why attention is described as content-based rather than position-based: it isn't reading distance, it's reading relevance.

In practice, the Transformer doesn't compute just one attention pattern per token — it computes several in parallel, called multi-head attention. Each "head" can learn to specialize in a different kind of relationship (one head might track grammatical subject/verb agreement, another might track pronoun references, another might track topical relevance), and their results are combined into the final representation.

3. Encoder/Decoder Structure

The original 2017 architecture is built from two stacks of layers working together:

The Encoder — reads the entire input sequence and, through repeated layers of self-attention, builds a rich contextual representation of it. Every token's representation is informed by every other token in the input.
The Decoder — generates the output sequence one token at a time, but with two sources of attention available at each step: self-attention over the output generated so far, and cross-attention back into the encoder's representation of the input.

Example: In a translation task ("The cat sat" → "Le chat s'est assis"), the encoder builds a full contextual representation of the English sentence. As the decoder generates each French word, cross-attention lets it look back at whichever parts of the English input are most relevant to the word it's producing right now — "chat" attends strongly back to "cat," "assis" attends back to "sat."

Component	Role	Typical use case
Encoder-only	Build understanding of input	Classification, embeddings (e.g., BERT)
Decoder-only	Generate output autoregressively	Chat, text generation (e.g., GPT-family models)
Encoder-decoder	Transform one sequence into another	Translation, summarization (e.g., T5)

Worth flagging for this site's purposes: most of the LLMs discussed elsewhere on this site — the assistants people actually talk to day to day — are decoder-only. They dropped the encoder half entirely and rely purely on self-attention over everything generated so far (the growing conversation) to decide what comes next. The full encoder/decoder split from the original paper is still the standard for translation-style tasks, but it isn't what's running under a typical chat assistant.

4. Why the 2017 Claim Held Up

The paper's title was a direct challenge to the prevailing wisdom of the time — that recurrence (RNNs) or convolution (CNNs) were necessary ingredients for handling sequences. The authors' claim was that attention alone, with no recurrence at all, was sufficient — and not just sufficient, but better.

"We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." — Vaswani et al., 2017

What made that bet pay off wasn't just architectural elegance — it was what the architecture unlocked downstream:

Parallel training at scale: Removing the sequential dependency meant training could fully exploit GPU/TPU hardware, processing an entire sequence's worth of tokens simultaneously instead of one at a time.
Bigger models became practical: Parallel training made it realistic to scale up both model size and training data by orders of magnitude — the direct lineage from the original Transformer to today's large language models.
Stronger long-range performance: Because every token attends directly to every other token (rather than through a chain of intermediate steps), long-range relationships stopped degrading the way they did in RNNs.

Nearly every model referenced elsewhere on this site — the LLM engine described in the How AI Actually Works series, the tokenization and neural network structure pages in this module — is a direct descendant of this architecture. The 2017 claim didn't just hold up; it became the default assumption for the entire field that followed it.