HAAW — From Pretraining to Post-Training

We Have Built a Base Model

Our training-data page ended with a numerical model whose weights have been shaped by enormous numbers of prediction-driven updates. That model can contain broad language capabilities. But what behavior should it produce when Craig types, “Explain this to me”?

PRETRAINING training sequences ↓ next-token objective ↓ loss → backpropagation → optimizer ↓ BASE MODEL

The base model has learned to continue token sequences. That alone does not fully specify the behavior of a helpful assistant.

Three Boxes We Should Not Mix Up

Base Model

A model produced by broad pretraining, commonly centered on prediction over large and varied datasets.

Post-Trained Model

A pretrained model whose parameters have undergone additional training toward further behavioral objectives.

Assistant System

The deployed model plus surrounding instructions, interface, runtime, tools, retrieval, memory systems, safety mechanisms and orchestration as applicable.

Model ≠ Assistant System. Some assistant behavior comes from learned weights. Some can come from the surrounding system at inference time.

Why Pretraining Alone Is Not the Whole Job

A base model learns statistical structure from its training objective. Given text, it can become very good at producing plausible continuations.

But a user often expects a different interaction:

USER: "Explain backpropagation without assuming I remember calculus." EXPECTED ASSISTANT BEHAVIOR: understand the request format follow the instruction answer the question choose useful detail avoid irrelevant continuation stop appropriately

Post-training can shape the model toward this kind of instruction-following and conversational behavior.

Stage 1 — Pretraining

Large and varied training sequences

↓

Tokenize and batch

↓

Predict subsequent tokens

↓

Calculate loss

↓

Backpropagate gradients

↓

Optimizer changes weights

↓

Repeat at enormous scale

The result is a base model whose parameters support broad learned capabilities and statistical regularities.

A Base Model Is a Continuation Machine

Suppose the input is:

Question: Why is the sky blue? Answer:

A sufficiently capable base model may continue this in an answer-like way because patterns of questions and answers occurred in training.

But that useful behavior emerged from sequence prediction. The base objective did not inherently say, “Always behave as a cooperative assistant for the human at the keyboard.”

Stage 2 — Instruction-Oriented Training

One common post-training approach supplies examples that explicitly demonstrate desired instruction-response behavior.

INSTRUCTION "Summarize this paragraph in two sentences." DESIRED RESPONSE [a concise two-sentence summary]

These examples are again training data. The machinery is familiar:

example → tokens → forward pass → loss → backpropagation → gradients → optimizer → changed weights

The objective and data have changed. The fundamental parameter-learning machinery has not.

Supervised Fine-Tuning

This kind of training is often called supervised fine-tuning, or SFT.

Pretraining

Broadly develops capabilities through large-scale predictive training.

SFT

Uses curated examples to further shape responses toward demonstrated tasks and interaction patterns.

“Fine-tuning” signals that we begin with an already trained parameter configuration rather than initializing the model from scratch.

The Same Matrices Continue to Change

pretrained parameter matrix W ↓ instruction-oriented training ↓ gradients ↓ optimizer ↓ post-trained parameter matrix W′

Post-training is not a layer of English instructions pasted onto the outside of the weights. When parameter training is involved, numerical tensors themselves are changed.

But Demonstrations Are Not the Only Signal

Developers may also want to distinguish between multiple possible model responses.

PROMPT ↓ candidate response A candidate response B ↓ preference / quality signal

A training process can use information about which responses are preferred according to a defined criterion. There are multiple families of techniques for doing this.

Preference-Based Post-Training

Human Feedback

People can evaluate, rank, compare or label model outputs according to specified criteria.

AI-Assisted Feedback

Other models or automated procedures can contribute evaluations or training signals under a designed process.

Direct Preference Methods

Some objectives train directly from preferred versus less-preferred responses without requiring a separate reinforcement-learning loop.

Reinforcement Methods

Other approaches use reward signals and reinforcement-learning algorithms to adjust model behavior.

There is no single universal post-training recipe.

RLHF — One Important Family

Reinforcement Learning from Human Feedback is a well-known approach, though it is not synonymous with all post-training.

Model produces candidate responses

↓

Humans provide preference information

↓

Preference signal is modeled or otherwise used

↓

Reinforcement-learning procedure updates model behavior

Specific implementations differ. HAAW should therefore resist the tempting but inaccurate equation:

POST-TRAINING ≠ ALWAYS RLHF

A Reward Is Not Applause Inside the GPU

“Reward” is another word that can invite anthropomorphism.

A reward signal is numerical training information.

The model is not necessarily experiencing approval or disappointment. The training algorithm uses numerical objectives to influence parameter updates.

response ↓ evaluation / reward-related signal ↓ optimization mathematics ↓ parameter changes

What Is Being Shaped?

Post-training can influence tendencies such as:

Instruction Following

Responding to requests in ways that better reflect the requested task.

Conversation

Participating in structured user/assistant interaction rather than merely continuing arbitrary text.

Response Style

Learned tendencies involving organization, relevance, tone and useful completion behavior.

Behavioral Constraints

Learned tendencies to handle certain categories of requests according to training objectives.

Exactly which properties are targeted and how successfully they are learned depends on the system and training procedure.

Capability and Behavior Are Related—but Not Identical

A useful distinction is:

CAPABILITY What computations / tasks can the learned model successfully perform? BEHAVIORAL TENDENCY Under what prompts and circumstances does it tend to perform them, and in what manner?

Post-training can change how capabilities are elicited or expressed without necessarily creating every underlying capability from scratch.

Post-Training Does Not Replace Pretraining

PRETRAINING builds broad learned structure ↓ POST-TRAINING further shapes behavior ↓ DEPLOYMENT surrounding system presents and controls access to model

The later stage depends on the learned foundation produced by the earlier one.

Now Add the Runtime System

Here is where a common misunderstanding becomes important.

ASSISTANT SYSTEM user interface ↓ system / application instructions ↓ conversation context ↓ possible memory / retrieval / tools ↓ serving and inference runtime ↓ POST-TRAINED MODEL ↓ generated output ↓ possible surrounding processing ↓ user interface

The deployed experience is the result of a stack, not merely a weight file.

System Instructions Are Not the Same as Training

An instruction supplied in the model's current context can strongly influence a response without changing the trained weights.

Post-Training

Changes model parameters through additional optimization.

Inference-Time Instruction

Changes the current context presented to the model; ordinarily does not rewrite trained weights.

This is the same boundary we established on the Context page: current information can change current behavior without becoming training.

Tools Are Not Learned Weights Either

An assistant may be able to search, calculate, retrieve a file, query a database or interact with an external service.

model / orchestrator determines useful action ↓ EXTERNAL TOOL ↓ tool returns result ↓ result enters model-visible context ↓ model continues generation

The external tool can provide capabilities and current information that do not reside in the LLM's parameters.

Persistent Memory Is Another Layer

past interaction ↓ external memory / storage ↓ selection or retrieval later ↓ current context ↓ model inference

A system that remembers something across conversations need not have retrained its LLM on that fact. Persistent memory can be stored outside the model and reintroduced when relevant.

Why This Matters

If we call every layer “the AI,” we lose the ability to explain what happened.

Was behavior caused by... learned pretrained weights? post-training? current system instruction? conversation context? retrieved information? persistent memory? tool result? sampling / generation settings? surrounding application logic?

Those are different mechanisms. WITNESS will eventually care very much about that distinction.

A Concrete Craig-to-Assistant Journey

CRAIG: "Explain gradients using my Calc III background." ↓ interface request + relevant context assembled ↓ tokenization ↓ post-trained model ↓ pretrained mathematical/language capability + post-trained instruction-following tendencies + current contextual information ↓ logits → selected tokens → stream ↓ ASSISTANT RESPONSE

No single box alone explains the complete interaction.

The Weights Still Do the Core Model Computation

We should not swing too far in the other direction. The surrounding system matters, but the model's learned parameters remain central.

current input representations + trained parameters ↓ transformer computation ↓ contextual representations ↓ logits ↓ tokens

Post-training has altered those parameters, so the model itself can acquire different response tendencies.

Training and Inference Meet

Pretraining data

↓

Base-model weights

↓

Post-training data / feedback / objectives

↓

Post-trained weights

↓

Deployment runtime + current context

↓

Assistant interaction

This is the bridge from our training pages back to the inference journey with which HAAW began.

How a Base Model Becomes an Assistant

Pretraining creates a broadly capable predictive model by changing weights across enormous amounts of training.

Post-training can further change those weights toward instruction-following and other desired behavioral objectives.

Preference and reinforcement techniques are possible post-training methods, but post-training is broader than any one technique.

At deployment, current instructions, tools, retrieval, memory and application logic can shape the assistant experience without necessarily changing the model weights.

The assistant a human encounters is therefore best understood as a system built around a trained model, not simply the LLM weight file by itself.

HAAW Checkpoint

Base model: A broadly pretrained model before later behavioral shaping stages.
Pretraining: Large-scale initial parameter training, commonly using predictive objectives such as next-token prediction.
Post-training: Additional training that modifies a pretrained model toward further behavioral objectives.
Supervised fine-tuning (SFT): Additional training using curated input/output examples or demonstrations.
Preference data: Information expressing that some candidate responses are preferred over others according to defined criteria.
RLHF: Reinforcement Learning from Human Feedback: a family of methods using human feedback in a reinforcement-learning training process.
Reward signal: Numerical information used by an optimization process; not evidence of a model experiencing reward.
Capability: A task or computation the learned model can successfully perform under suitable conditions.
Behavioral tendency: How the model tends to respond or express its capabilities under particular inputs.
Inference-time instruction: An instruction supplied in current context; it can alter current behavior without ordinarily changing trained weights.
Assistant system: The deployed model plus the surrounding interface, runtime, context assembly, tools, retrieval, memory and other orchestration as applicable.
Central lesson: Pretraining builds the base model; post-training shapes the model further; deployment wraps that model in a larger interactive system.

Next HAAW Page: Evaluation — How Do We Know Whether Training Actually Worked?