Tokens Become Vectors

Where We Left the Prompt

Human-readable text has become an ordered sequence of integer token IDs.

The actual token boundaries and ID numbers depend on the tokenizer used by the model.

But an ID is merely a vocabulary label. If one token has ID 500 and another has ID 1000, the second token is not twice as meaningful, twice as large, or twice as important.

The Model Needs Something It Can Calculate With

A transformer performs mathematical operations on arrays of numbers. It therefore needs a richer numerical representation for every token in the input sequence.

That representation is called an embedding vector.

A vector is simply an ordered list of numbers. In a modern language model it may contain hundreds or thousands of numerical components.

The example is intentionally tiny. Real embedding vectors are generally far larger.

Where Does the Vector Come From?

The model contains a learned embedding table, sometimes called an embedding matrix.

Conceptually, there is an embedding vector associated with each token in the model's vocabulary. During training, these numerical values are learned along with the model's other parameters.

The token ID tells the embedding operation which learned row/vector corresponds to that token.

1. Embedding Lookup

For every token ID in the prompt, the model retrieves the corresponding learned embedding vector.

This is often described as a lookup, although implementations may realize the operation through optimized tensor operations.

2. Every Token Gets a Vector

If the prompt contains many tokens, each token position receives a vector.

Together, the vectors form a two-dimensional array: one row per token position and one column per embedding dimension.

Now the prompt has become a large block of numbers suitable for neural-network computation.

What Do the Numbers Mean?

This is where casual explanations can become misleading.

It is tempting to imagine individual dimensions as neat human labels such as:

That is generally not a reliable description of how learned representations work.

The representation is distributed across many dimensions. Relationships useful to the model emerge from patterns across the vector and from later transformations through the network.

But We Have a Problem: Order

Embedding lookup tells the model which token occupies each row, but a transformer also needs information about where tokens occur in the sequence.

The token identities may be the same while their positions differ dramatically.

Different transformer architectures represent position in different ways. Some use learned or fixed positional embeddings; many modern models use mechanisms such as rotary positional embeddings (RoPE) within attention.

Position Is Not Another Word

Position information does not mean inserting words such as “first,” “second,” and “third” into the prompt.

It is represented mathematically so later operations—especially attention—can distinguish relationships among tokens at different positions.

The exact way these are combined depends on the model architecture, so we will not force all transformers into one implementation.

A Crucial Distinction: Initial vs. Contextual Representation

The embedding vector is only the starting representation for a token.

At this point, a token's representation has not yet been transformed by all the surrounding tokens through the stack of transformer layers.

Consider the word:

In “river bank” and “bank account,” the same token may begin from the same learned token embedding. Its representation becomes context-sensitive as transformer computations incorporate relationships with other tokens.

Where Is This Happening Physically?

We should not lose sight of our machine.

The token IDs and embedding data are numerical arrays in computer memory. The accelerator executes operations using those arrays and the model parameters loaded earlier.

There is no separate little language engine hidden inside the GPU. At the hardware level, we still have memory, numerical data, instructions, and enormous amounts of arithmetic.

Our Journey So Far

HAAW Checkpoint

Established mechanism

Token IDs are mapped to learned embedding vectors used as numerical representations.

Established mechanism

Embedding parameters are learned during training and are part of the trained model.

Important distinction

A token ID is an identifier; an embedding is a learned vector.

Important distinction

The initial token embedding is not the token's final context-sensitive representation.

Architecture-dependent

Transformers use different mechanisms to represent position, including positional embeddings and rotary positional methods.

Interpretive caution

Embedding dimensions should not generally be treated as individually named human concepts.

Still ahead

The first transformer layer and the mechanism by which attention relates token positions to one another.