How AI's Actually Work — HAAW

Tensors and Matrix Multiplication

The Arithmetic Underneath — The GPU is fast, but fast at what? Now we open the numerical objects and operations that transformer diagrams have been hiding behind letters.

Everything Has Become Numbers

By the time our prompt reaches the transformer, the GPU is not manipulating English words. It is operating on organized collections of numerical values.

Token representations are numbers. Model weights are numbers. Attention state is numbers. Activations are numbers. Logits are numbers.

The organization of those numbers matters.

AI software stores them in numerical arrays with defined shapes. We commonly call those arrays tensors.

Build Up to a Tensor

Scalar

One value.

7.4

Vector

A one-dimensional ordered list.

[ 2.1, -0.7, 4.3 ]

Matrix

A two-dimensional array arranged in rows and columns.

Tensor

The general multidimensional numerical array used throughout machine-learning software.

In practice, software libraries often use the word tensor for arrays even when they happen to be one- or two-dimensional.

A Matrix Is Just Numbers With Shape

Here is a small 2 × 3 matrix:

A = 123 456

shape(A) = 2 rows × 3 columns

The values matter, but so does the shape. The dimensions determine which mathematical operations are valid and how data moves through the model.

Why Matrices Appear Everywhere in an LLM

A transformer repeatedly takes one collection of values and transforms it into another using learned parameters.

Those learned parameters are commonly organized into weight matrices.

input values × learned weight matrix → new values

That pattern appears again and again in embeddings, attention projections, feed-forward networks, and the final vocabulary projection.

Matrix Multiplication — One Tiny Example

The name sounds formidable. The basic operation is much less mysterious.

Take a row from the first matrix and a column from the second. Multiply matching values, then add those products.

[ 1   2   3 ]  ·  [ 4, 5, 6 ]ᵀ

= (1 × 4) + (2 × 5) + (3 × 6)
= 4 + 10 + 18
= 32

That result becomes one entry in the output matrix.

Matrix multiplication scales this same idea across many rows and columns.

Real LLM matrices can be enormous, but the arithmetic is built from operations of this general kind: multiplication and accumulation performed at huge scale.

Why the Shapes Must Fit

For ordinary matrix multiplication, the inside dimensions must agree.

(2 × 3) × (3 × 4) → (2 × 4)

The three values across each row of the first matrix pair with the three values down each column of the second.

The resulting matrix has the outside dimensions: 2 × 4.

Now Return to the Transformer

We previously wrote:

Q = X × WQ
K = X × WK
V = X × WV

Now those letters have physical meaning.

X

A tensor containing the current token representations entering that operation.

WQ

A learned weight matrix used to transform those representations into queries.

WK

A learned weight matrix used to produce keys.

WV

A learned weight matrix used to produce values.

The GPU is performing large numerical operations that implement these transformations.

The Weights Are the Learned Numbers

Before training, model parameters do not yet contain the learned structure produced by training.

Training repeatedly adjusts parameter values according to an optimization process.

After training, those learned numerical values are stored as the model's weights.

Training adjusts parameter values
↓
Learned weights are saved
↓
Inference loads those weights
↓
GPU uses them in tensor operations

During ordinary inference, the model is primarily using its trained weights, not retraining them with every prompt.

Weights Versus Activations

Weights

Learned model parameters. They persist in the saved model and are reused across requests unless the model itself is changed.

Activations

Intermediate numerical results produced while a particular input is moving through the model.

input activation
    ×
model weights
    ↓
new activation
    ↓
next operation / next layer

One Prompt Creates a River of Tensors

Our simple question—“Why is the sky blue?”—does not become one permanent numerical object and sit still.

Representations are repeatedly transformed as they pass through the network.

Token IDs
↓
Embedding tensors
↓
Q / K / V tensors
↓
Attention results
↓
MLP activations
↓
Next transformer layer
↓
Final hidden representation
↓
Vocabulary logits

Each stage produces numerical state used by the next.

Why the GPU Loves This

A large matrix operation contains many individual calculations that can be organized for parallel execution.

Thousands / millions / billions of suitable arithmetic operations
        ↓
divide work across accelerator resources
        ↓
many multiply-and-accumulate operations execute concurrently
        ↓
output tensor

This is why matrix-heavy neural-network computation maps so effectively onto GPU and other accelerator architectures.

Precision: How Many Bits Per Number?

Not every model value must necessarily be represented using the same large general-purpose numerical format.

AI systems may use formats such as FP32, FP16, BF16, FP8, or quantized integer-like representations, depending on hardware, model design, accuracy requirements, and optimization choices.

Smaller numerical representations can reduce memory use and data movement and may increase computational throughput.

But reduced precision involves engineering tradeoffs. It is not simply “smaller is always better.”

This Explains Some of the Speed

We now have several pieces working together:

Parallel Hardware

Many suitable arithmetic operations can execute concurrently.

High-Bandwidth VRAM

Large amounts of numerical model data can be supplied rapidly to accelerator computation.

Specialized Matrix Units

Modern accelerators include hardware optimized for common AI matrix operations.

Compact Number Formats

Appropriate lower-precision representations can reduce the cost of storing, moving, and processing model data.

So while we watch a sentence appear on screen, the hardware can be performing an extraordinary number of simple numerical operations underneath it.

The Arithmetic Underneath

The transformer is sophisticated in its organization, learned parameters, and scale.

But when we descend far enough, we do not find little English concepts moving around the GPU.

We find numbers in memory, arranged into tensors, repeatedly transformed through operations that include vast amounts of multiplication and addition.

Meaning is visible to us in the behavior of the complete trained system. The GPU itself is executing arithmetic.

HAAW Checkpoint

Tensor
A multidimensional numerical array used to hold model parameters, inputs, activations, attention state, logits, and other data.
Matrix multiplication
A structured operation combining rows and columns through repeated multiply-and-add calculations.
Shape
The dimensions of a tensor; shape determines how data is organized and which operations are compatible.
Weight
A learned parameter value stored as part of the trained model.
Activation
An intermediate numerical value produced while a particular input is processed.
Inference
Uses the trained weights to transform input through the model; ordinary inference does not mean retraining those weights for each prompt.
Physical reality
These tensor operations become memory transfers and arithmetic executed by real processor hardware.