HAAW — Backpropagation

We Ended With a Question

A model produces an output. Training calculates a loss. But the model may contain billions of trainable numbers. Which ones should change—and by how much?

billions of parameters
    │
    ▼
model computation
    │
    ▼
LOSS

Question:
How did each trainable parameter affect this loss?

Backpropagation provides an efficient answer in the form of gradients.

First: What Backpropagation Is NOT

Not Reverse Language

The model does not generate the sentence backward.

Not Blame

Weights are not assigned moral or semantic “fault.” Derivatives measure numerical sensitivity.

Not the Optimizer

Backpropagation computes gradients. The optimizer uses those gradients to update parameters.

Not a Second Model

It is an algorithmic use of the same computation's mathematical relationships.

Start With One Tiny Chain

Forget transformers for a moment. Suppose:

w = 2
x = 3

y = w × x
L = y²

Forward:

y = 2 × 3 = 6
L = 6² = 36

We now ask: if w changed slightly, how would the loss L change?

Work Backward One Relationship at a Time

We have:

w → y → L

The derivative of the loss with respect to y is:

dL/dy = 2y = 12

And y = wx, so:

dy/dw = x = 3

The chain rule connects them:

dL/dw = dL/dy × dy/dw

dL/dw = 12 × 3

dL/dw = 36

We have now calculated the local sensitivity of the loss to the parameter w.

What Does 36 Mean?

At this point in the calculation, the derivative is positive.

Locally, increasing w would increase the loss; decreasing w would decrease it.

A simple gradient-descent optimizer could therefore move w in the opposite direction of the gradient:

w_new = w_old − η(dL/dw)

If the learning rate η were 0.01:

w_new = 2 − 0.01(36)
w_new = 1.64

Backpropagation found the gradient. The optimizer performed the update.

Why the Chain Rule Matters

Real neural networks contain long chains and branching networks of operations.

w
│
▼
operation A
│
▼
operation B
│
▼
operation C
│
▼
loss

If each stage has a known derivative relationship, the chain rule lets us combine those local relationships to determine how an early value influences the final loss.

∂L/∂w =
∂L/∂C × ∂C/∂B × ∂B/∂A × ∂A/∂w

Backpropagation organizes this calculation efficiently across the network.

A Computation Graph

A useful mental model is a graph of operations and values.

FORWARD

X ──┐
   ├──→ [ MATRIX MULTIPLY ] → A → [ NONLINEARITY ] → B ──┐
W ──┘                                                 │
Target ───────────────────────────────────────────────┤
                                                      ▼
                                                [ LOSS ]

During the forward pass, values are computed through this graph. During the backward pass, derivative information is propagated through the graph in reverse dependency order.

The “Error” Does Not Crawl Backward Like a Packet

The title “following the error back” is useful shorthand, but we should make the physical picture accurate.

There is not one little object called ERROR traveling backward through the GPU.

The training software executes mathematical operations that calculate derivatives and gradients using relationships established by the forward computation.

What moves through memory and accelerator hardware are numerical tensors and instructions involved in those derivative calculations.

Now Replace Scalars With Matrices

Our tiny example used one weight. A neural network uses tensors containing many values.

Y = XW

During the forward pass, matrix multiplication transforms X using W.

During backpropagation, the system needs gradients such as:

∂L/∂W gradient for the weight matrix
∂L/∂X gradient flowing toward earlier computation

These gradient calculations themselves become large matrix operations—the kind of highly parallel arithmetic GPUs are designed to execute efficiently.

A Familiar Matrix Pattern Appears

For a simple matrix multiplication Y = XW, the backward calculations involve transposed matrices.

Given upstream gradient: G = ∂L/∂Y

gradient for W:
∂L/∂W = XᵀG

gradient for X:
∂L/∂X = GWᵀ

This is a simplified but very real glimpse of why matrix operations dominate both the forward and backward passes.

“Upstream Gradient”

When backpropagation reaches an operation, it receives information describing how the final loss changes with respect to that operation's output.

That incoming derivative information is often called the upstream gradient.

later computation / loss
    │
    ▼
upstream gradient
    │
    ▼
current operation
    │
    ├──→ gradient for local parameters
    └──→ gradient for earlier inputs

Each operation contributes its local derivative relationships, and the chain rule connects them.

Back Through a Transformer Block

A transformer block contains several operations with dependencies:

input X
  │
  ├─→ normalization
  ├─→ Q, K, V projections
  ├─→ attention scores
  ├─→ softmax / attention weighting
  ├─→ output projection
  ├─→ residual addition
  ├─→ MLP transformations
  └─→ residual addition
      │
      ▼
updated representation

During training, gradients must be calculated through the differentiable operations that contributed to the loss, including gradients for the trainable projection and feed-forward weights.

Residual Connections Create Multiple Paths

A residual connection combines an earlier representation with a transformed version of it.

X ───────────────────────┐
│                       │
└→ transformation → F(X) ┤
                        ▼
            X + F(X)

During backpropagation, gradients follow the dependency paths created by the computation graph and contributions from branches are combined appropriately.

This is one reason “backward” should not be imagined as one single straight line.

Softmax and Attention Are Differentiable Too

Recall attention:

Attention(Q,K,V) = softmax(QKᵀ / √d) V

During training, derivative calculations propagate through these operations as well.

Loss gradient

↓

Attention output

↓

Weighted values

↓

Softmax / attention scores

↓

Q, K and V

↓

Projection weight gradients

The formulas become more involved, but the organizing principle remains the chain rule.

Why Save Forward-Pass Information?

Backward calculations often need values that were produced during the forward pass.

FORWARD
input → [operation] → activation → [operation] → activation → loss
│ │
└──── values useful later ─────┘

BACKWARD
loss derivatives → gradients through those operations

Keeping activations consumes memory. Recomputing some of them later consumes additional computation. Training systems can trade one resource for the other.

Automatic Differentiation

Modern machine-learning frameworks generally do not require a programmer to hand-derive and hand-code every derivative in a giant transformer.

They use automatic differentiation: software records or constructs enough information about mathematical operations to systematically compute the needed derivatives.

program expresses forward operations
    │
    ▼
framework tracks differentiable relationships
    │
    ▼
backward computation constructed / executed
    │
    ▼
parameter gradients

For neural-network training, reverse-mode automatic differentiation is especially useful because one scalar loss depends on very many parameters.

Why Reverse Mode Fits Neural Networks

Training often has:

BILLIONS OF INPUT PARAMETERS
↓
large computation
↓
ONE SCALAR LOSS

We want the derivative of that one loss with respect to a huge number of parameters.

Reverse-mode differentiation can efficiently reuse intermediate derivative results while working backward from the scalar objective.

Gradient Accumulation

A weight participates in computations involving many training tokens and examples. Contributions to its gradient must be combined appropriately.

example / token contributions
    ├──→ gradient contribution
    ├──→ gradient contribution
    ├──→ gradient contribution
    └──→ gradient contribution
         │
         ▼
combined gradient used for update

Training implementations may also accumulate gradients across multiple smaller microbatches before performing an optimizer step, which can help when accelerator memory limits the batch size processed at once.

Across Multiple GPUs

Distributed training adds another physical problem: gradients or related training state may need to be communicated among devices, depending on the parallelism strategy.

GPU 0 computes local work ─┐
GPU 1 computes local work ─┼──→ COMMUNICATION / COLLECTIVE OPERATIONS
GPU 2 computes local work ─┤
GPU 3 computes local work ─┘
                               │
                               ▼
                      coordinated parameter update

At scale, backpropagation is therefore both a mathematical problem and a data-movement problem.

The Hardware View

Nothing supernatural occurs when the model “learns.”

VRAM contains:
├── parameter tensors
├── activation tensors
├── gradient tensors
├── optimizer-related state
└── temporary buffers

GPU executes:
├── matrix multiplications
├── elementwise operations
├── reductions
├── derivative computations
└── parameter-update-related arithmetic

Electrical activity in hardware changes numerical values stored in memory according to software executing mathematical algorithms.

One Complete Learning Step

Training batch enters

↓

Forward pass produces logits

↓

Loss is calculated

↓

Backward pass calculates gradients

↓

Optimizer reads gradients

↓

Parameters are updated

↓

Next training step begins

↺

And Then Millions or Billions of Nudges

No individual update needs to create something recognizable as grammar, factual association, style, or reasoning behavior.

Useful model behavior emerges from the accumulated effect of enormous numbers of parameter adjustments across training.

small numerical update
+ small numerical update
+ small numerical update
+ ...
+ vast numbers of additional updates
────────────────────────────
trained parameter configuration

We should be careful with the word “emerges”: it describes the observed result of the optimization process, not an unexplained extra mechanism.

Following the Error Back

We can now make the title precise.

The forward pass creates the computation that leads to the loss.

Backpropagation traverses those mathematical dependencies in reverse order.

The chain rule converts local derivative relationships into gradients of the final loss with respect to trainable parameters.

The optimizer then uses those gradients to change the parameters.

Backpropagation does not teach the model by explaining its mistake. It supplies the numerical information that makes iterative optimization possible.

HAAW Checkpoint

Backpropagation: An efficient method for calculating gradients of a loss with respect to model parameters by applying the chain rule through the computation graph in reverse dependency order.
Derivative: A mathematical description of how one quantity changes locally with respect to another.
Gradient: A collection of partial derivatives describing the local sensitivity of the loss to many parameters.
Chain rule: The calculus rule that connects derivatives through composed functions.
Computation graph: A representation of values and mathematical operations showing the dependencies created during computation.
Upstream gradient: Derivative information arriving at an operation from later computations during the backward pass.
Automatic differentiation: Software techniques that systematically derive and execute derivative calculations from mathematical operations.
Reverse-mode differentiation: A differentiation strategy especially effective when one scalar output, such as loss, depends on many input parameters.
Gradient accumulation: Combining gradient contributions before an optimizer update, sometimes across multiple microbatches.
Backpropagation versus optimizer: Backpropagation computes gradients. The optimizer decides how those gradients change parameter values.

Next HAAW Page: The Optimizer — Turning Gradients Into Better Weights