HAAW — The Optimizer

We Have the Gradient. Now What?

A gradient gives local directional information. It does not, by itself, specify the complete training strategy.

forward pass → loss → backpropagation → GRADIENTS
                                           │
                                           ▼
OPTIMIZER
                                           │
                                           ▼
UPDATED PARAMETERS

The optimizer is the rule—or family of rules—that converts gradient information into parameter updates.

Three Questions

Which Direction?

The gradient tells us the local direction in parameter space in which the loss increases most rapidly. Minimization generally moves against it.

How Far?

The learning rate and optimizer mechanics determine the scale of the update.

What History Matters?

Modern optimizers can use accumulated information from earlier gradients to influence current updates.

Start With Plain Gradient Descent

For a parameter w, the simplest update is:

w_new = w_old − η ∂L/∂w

η = learning rate

For an entire parameter vector or tensor collection, the same idea is written compactly as:

θ_new = θ_old − η ∇L(θ)

Here θ represents the model's trainable parameters and ∇L is the gradient of the loss with respect to them.

Craig's Calc III Gradient Has Arrived

In multivariable calculus, a scalar function can depend on several variables:

L = L(w₁, w₂, w₃)

Its gradient is:

∇L = [ ∂L/∂w₁, ∂L/∂w₂, ∂L/∂w₃ ]

An LLM training objective is the same kind of mathematical idea taken to an extreme scale:

L = L(w₁, w₂, w₃, ... , wₙ)

where n may be enormous

The dimensionality became outrageous. The calculus did not become a different species.

A Loss Landscape

For two hypothetical parameters, we could imagine loss as a surface over a plane.

higher loss
/\
________/ \______
__/ \__
__/ \__
/ ● \
/ current point \
/ \
/____________ lower region ____________\

parameter w₁ parameter w₂

The gradient describes the local uphill direction. Moving against the gradient gives a local downhill direction.

With billions of parameters, we cannot draw the surface, but optimization still operates in that high-dimensional parameter space.

The Learning Rate Is the Step Size

The learning rate η determines how strongly the optimizer responds to gradient information.

Too Large

Updates may overshoot useful regions, oscillate, or destabilize training.

Too Small

Training may progress very slowly or struggle to move effectively through the landscape.

Useful Scale

Updates can make productive progress while remaining sufficiently stable.

In real training, the learning rate commonly changes over time according to a schedule rather than remaining one constant value from beginning to end.

Why Not Use the Entire Dataset for Every Update?

Computing an exact gradient over an enormous training dataset before every parameter update would be extremely expensive.

Instead, neural-network training commonly uses batches or minibatches of examples.

huge training dataset
    │
    ├──→ batch 1 → gradient → update
    ├──→ batch 2 → gradient → update
    ├──→ batch 3 → gradient → update
    └──→ ...

The resulting gradient is an estimate based on the current batch rather than an exact full-dataset gradient.

Stochastic Gradient Descent

Stochastic gradient descent, or SGD, refers to optimization using gradient estimates from subsets of the data rather than requiring the complete dataset for each update.

In modern deep learning, people often use “SGD” broadly while actual implementations use minibatches and may add momentum or other refinements.

The gradient path becomes noisy—but computationally practical.

That noise is not necessarily a defect. Optimization does not require every update to be the exact full-dataset downhill direction.

Momentum: Remember Some Direction

If successive gradients repeatedly point in similar directions, it can be useful to accumulate some of that directional history.

gradient step 1 →
gradient step 2 →
gradient step 3 →
gradient step 4 →

persistent direction can build momentum

A simplified momentum idea maintains a running update-related quantity:

v_t = βv_(t−1) + g_t

θ_t = θ_(t−1) − ηv_t

Exact sign conventions and formulations vary, but the central idea is that the optimizer can use more than the current gradient alone.

Why History Helps

Imagine descending a narrow valley where one direction slopes steadily toward a better region while another causes repeated side-to-side oscillation.

\ ↘ ↙ /
\ ↘↙ /
\ ↙↘ /
\ ↘ ↙ /
\ ↓ /
\ ↓ /
\ ↓ /
\_____/

Momentum-like methods can damp some oscillatory behavior while preserving persistent movement.

Different Parameters Can Need Different Effective Step Sizes

A single global learning rate does not mean every parameter necessarily benefits from identical update behavior.

Adaptive optimizers use statistics of past gradients to scale updates differently for different parameters.

parameter A: gradient history → effective scaling A
parameter B: gradient history → effective scaling B
parameter C: gradient history → effective scaling C

This brings us to one of the most familiar optimizer families in transformer training.

Adam: Two Running Estimates

Adam maintains moving estimates related to:

First Moment

A moving average of gradients—roughly tracking persistent gradient direction.

Second Moment

A moving average of squared gradients—tracking gradient magnitude information.

In simplified notation:

m_t = β₁m_(t−1) + (1−β₁)g_t

v_t = β₂v_(t−1) + (1−β₂)g_t²

Adam also uses bias corrections for these moving estimates, especially early in optimization.

The Adam Update

After bias correction, a simplified Adam-style parameter update can be written:

m̂_t = m_t / (1−β₁ᵗ)
v̂_t = v_t / (1−β₂ᵗ)

θ_t = θ_(t−1) − η · m̂_t / (√v̂_t + ε)

The details matter in real implementations, but the conceptual structure is enough for HAAW:

Current gradient

↓

Update running gradient statistics

↓

Correct / scale those estimates

↓

Compute parameter update

AdamW and Weight Decay

Modern transformer training often uses optimizer variants such as AdamW.

AdamW separates weight decay from Adam's gradient-based adaptive update in a way that differs from simply adding conventional L2 regularization inside the loss gradient.

gradient-based adaptive update
+
separately applied weight-decay behavior
=
AdamW-style parameter update

The exact optimizer choice and hyperparameters vary among training systems. HAAW's important point is that “the optimizer” is a specific numerical algorithm, not merely a synonym for backpropagation.

The Optimizer Has Memory—But Not Human Memory

Here is another place where the word “memory” can mislead.

Adam-family optimizers maintain numerical state from previous optimization steps.

for many trainable parameters:

parameter value θ
gradient g
first-moment state m
second-moment state v

That state lets the optimizer use gradient history. It does not mean the optimizer remembers sentences, concepts, or training examples as semantic memories.

Optimizer State Costs Real Memory

Those extra numerical tensors must live somewhere.

ACCELERATOR / TRAINING MEMORY

├── model parameters
├── gradients
├── activations
├── optimizer first-moment state
├── optimizer second-moment state
└── temporary / runtime buffers

Depending on precision choices and training strategy, optimizer-related state can represent a substantial memory cost.

The optimizer is mathematical state with a physical address.

Its running statistics ultimately become arrays of numbers occupying RAM, VRAM, or distributed storage/memory according to the training architecture.

Precision Complicates the Picture

Training does not necessarily perform every operation or store every value in one numerical format.

Modern systems may use mixed-precision techniques—for example, lower-precision formats for much of the expensive tensor computation while retaining higher-precision state where needed for numerical stability or parameter updates.

fast lower-precision tensor work
        │
        ├──→ training computation
        │
higher-precision state where needed
        └──→ stability / updates

Exact precision strategies vary by hardware, framework, optimizer, and model.

Learning-Rate Schedules

The useful step size at the beginning of training may not be the useful step size later.

Training systems therefore often change the learning rate according to a schedule.

training begins
    │
    ├──→ possible warmup
    │
    ├──→ larger / scheduled learning region
    │
    └──→ gradual decay
         ↓
training ends

Warmup and decay are common ideas, but exact schedules differ widely.

Gradient Clipping

Occasionally gradients can become undesirably large. Some training systems limit their magnitude before applying an update.

computed gradient
    │
    ▼
check magnitude / norm
    │
    ├── acceptable → use normally
    └── too large → scale / clip

This is another example of the optimizer/training system doing more than blindly subtracting a raw gradient.

One Parameter Update Is Tiny

A training step generally does not rewrite a model into a completely different machine.

old weight: 0.173842...
update: −0.0000...
new weight: 0.1737...

Multiply that idea across enormous parameter arrays and enormous numbers of training steps.

The trained model is the accumulated result.

No One Assigns a Meaning to Each Weight

The optimizer is not told:

weight 7,294,102 = "sky"
weight 7,294,103 = "blue"
weight 7,294,104 = "scattering"

Instead, it repeatedly applies numerical update rules across distributed representations.

Useful language behavior arises from the combined configuration and interactions of many learned parameters.

Distributed Optimizers

When a model is trained across many accelerators, optimizer state itself may become too large or expensive to replicate everywhere.

Large-scale systems can partition parameters, gradients, and optimizer state across devices and coordinate updates through communication.

GPU 0 ─ parameter / gradient / optimizer shards ─┐
GPU 1 ─ parameter / gradient / optimizer shards ─┤
GPU 2 ─ parameter / gradient / optimizer shards ─┼→ coordinated training
GPU N ─ parameter / gradient / optimizer shards ─┘

Once again, mathematics meets topology, memory capacity, bandwidth, and synchronization.

The Whole Training Loop Now Has Its Parts

Training data / batch

↓

Forward pass

↓

Loss

↓

Backpropagation

↓

Gradients

↓

Optimizer

↓

Updated weights

↓

Next batch

↺

That loop—repeated at tremendous scale—is the engine that turns an initialized model into a trained model.

Turning Gradients Into Better Weights

The gradient supplies local information about the loss landscape.

The learning rate controls update scale.

Momentum and adaptive methods can incorporate information from previous gradients.

Optimizer state occupies real memory and participates in real numerical computation.

The optimizer changes parameter values; it does not understand the semantic meaning of individual weights.

Training is not a teacher whispering knowledge into a model. It is an enormous iterative numerical optimization process.

HAAW Checkpoint

Optimizer: The numerical algorithm that converts gradient information into updates of trainable parameters.
Gradient descent: A basic optimization method that moves parameters in the direction opposite the local loss gradient.
Learning rate: A hyperparameter controlling the scale of parameter updates.
SGD: Stochastic gradient descent: optimization using gradient estimates from subsets or minibatches of training data.
Momentum: An optimization technique that incorporates a running history of gradients or update direction.
Adam: An adaptive optimizer using moving estimates related to the first and second moments of gradients.
AdamW: An Adam-family optimizer that decouples weight decay from the adaptive gradient update.
Optimizer state: Additional numerical values maintained across training steps, such as moving gradient statistics.
Learning-rate schedule: A rule that changes the learning rate as training progresses.
Gradient clipping: Limiting gradient magnitude to reduce the effect of excessively large updates.
Optimization landscape: A conceptual high-dimensional surface defined by loss as a function of model parameters.
Central lesson: Backpropagation computes gradients; the optimizer turns those gradients and its state into new parameter values.

Next HAAW Page: Training Data — What the Model Learns From and What “Learning” Really Means