HAAW — Evaluation

The Optimizer Says the Numbers Improved

Eventually we must stop watching the training machinery and ask: what can the resulting model actually do?

TRAINING data → prediction → loss → gradients → optimizer → changed weights EVALUATION new test conditions → model behavior → measurements / judgments

Evaluation is where claims about a trained model meet evidence.

Training Loss Is Not the Final Exam

A falling training loss says the model is improving according to its training objective on training data. That is useful evidence, but it is not enough.

A model can fit its training objective better without performing equally well on new examples or real tasks.

Craig's Multiplication Test

training example: 17 × 24 → 17 × (20 + 4) → 340 + 68 → 408

If we later ask exactly 17 × 24 and receive 408, perhaps the model learned a reusable procedure—or perhaps that example was memorized. One test cannot distinguish them.

stronger evaluation: 23 × 16 → 23 × (10 + 6) → 230 + 138 → 368

Success on genuinely new problems is stronger evidence of generalization.

Train, Validate, Test

Training Set

Data used to drive parameter updates.

Validation Set

Held-out data used during development to monitor or compare choices.

Test Set

Held-out data intended for a more independent estimate after development decisions.

The key discipline is separation: do not judge generalization only on material used to fit the parameters.

Contamination

LLMs complicate the idea of a clean exam because their training corpora can be enormous.

benchmark question ↓ Was it unseen? Was a near-duplicate seen? Was its answer discussed elsewhere? Was the benchmark publicly available? ↓ CONTAMINATION QUESTION

Data provenance and contamination matter when interpreting benchmark scores.

A Benchmark Is a Measuring Instrument

Knowledge

Domain questions testing recall or application.

Mathematics

Numerical, symbolic, or multi-step problems.

Code

Generated programs checked by tests or other criteria.

Reasoning

Tasks intended to probe relationships, inference, planning, or problem solving.

No single benchmark measures “intelligence” in its entirety.

How Do We Score It?

Exact Match

Does the output equal the expected answer?

Programmatic Check

Can code, math, or structured output be tested automatically?

Human Evaluation

People judge qualities difficult to reduce to one automatic rule.

Model-Based Evaluation

Another model can apply a rubric, with its own limitations.

Correct Answer, Wrong Reason?

QUESTION: 23 × 16 MODEL ANSWER: 368

The final answer is correct. But if we are evaluating a reasoning procedure, the answer alone does not establish how the internal computation produced it.

Observed output is evidence of behavior. It is not automatically a complete observation of internal mechanism.

A Good Explanation Is Not Necessarily an Internal Trace

model computation ↓ answer + generated explanation DO NOT AUTOMATICALLY EQUATE: generated explanation = complete internal computation trace

A coherent explanation can be useful while still being a generated output rather than a literal record of every internal operation.

Evaluate the Claim You Actually Care About

CLAIM: "The model can do X." ↓ design an evaluation that actually tests X ↓ collect results ↓ make a bounded evidence-based claim

If we care about arithmetic generalization, memorized facts are weak evidence. If we care about factual reliability, eloquent prose is not enough.

Evaluation Is Multi-Dimensional

Capability

Can it perform the task?

Reliability

How consistently?

Generalization

Does success transfer to new cases?

Behavior

Does it follow instructions appropriately?

Robustness

What happens under difficult or altered conditions?

Efficiency

What compute, memory, latency, and throughput are required?

Regression: Did Something Else Get Worse?

Post-training may improve one behavior while degrading another. Regression testing checks whether useful prior behavior was damaged.

MODEL A → math ✓ coding ✓ instruction following ~ MODEL B → math ✓✓ coding ? instruction following ✓ "Better" is rarely one-dimensional.

Overfitting the Exam

If developers repeatedly inspect one benchmark and change the system to improve its score, the benchmark can gradually become part of the development loop.

benchmark → score → modify system ↑ ↓ └──── repeat ──────┘ Are we improving the capability, or performance on this particular exam?

Model Evaluation vs System Evaluation

Model Evaluation

Tests properties of trained weights under defined input and generation conditions.

System Evaluation

Tests the deployed combination of model, instructions, runtime, retrieval, tools, memory, interface, and other components.

This continues our earlier boundary: LLM ≠ complete assistant system.

The Physical Evaluation Loop

evaluation cases on storage ↓ runtime loads cases ↓ CPU / GPU inference ↓ generated outputs / scores ↓ evaluation software and/or human judgments ↓ recorded results

Evaluation is itself a workload: data, computation, outputs, measurements, and evidence.

WITNESS Raises Its Hand

EVALUATION: What did the system do, and how well? WITNESS: What observable events occurred inside and around the system while it did it?

Performance evidence and mechanism evidence are different. Together they can support a richer examination.

How Do We Know Training Worked?

Not merely because training loss went down.

We test the resulting model under carefully chosen, preferably held-out conditions.

We distinguish memorization from evidence of generalization.

We measure the capabilities and behaviors we actually claim to care about.

We recognize contamination, overused benchmarks, regressions, and incomplete measurements.

And we distinguish evaluating the model from evaluating the complete assistant system.

HAAW Checkpoint

Evaluation: Systematic measurement or judgment of model or system behavior under defined conditions.
Training loss: A training-objective measure; not by itself proof of generalization.
Validation set: Held-out data used during development.
Test set: Held-out data intended for a more independent final estimate.
Generalization: Successful behavior on inputs not identical to training examples.
Benchmark: A defined task collection and scoring procedure.
Contamination: Training exposure that compromises the independence of evaluation material.
Metric: A quantitative performance measurement.
Regression: Loss of previously useful behavior after a change.
Model evaluation: Testing the trained model under controlled conditions.
System evaluation: Testing the larger deployed stack.
Central lesson: A lower loss tells us optimization changed the model. Evaluation asks whether the resulting behavior supports the claims we want to make.

Next HAAW Page: How Does a Model Learn to Reason? — From Worked Examples to Reusable Procedures