We Opened the Machine
On the previous page we followed a reasoning-like step down into token representations, attention, MLPs, residual paths, activations, logits and the next-token loop. Now comes the uncomfortable question: if we can see those numbers, do we understand them?
Observability Is Not Interpretability
Observability
We can capture or measure an internal event, state, tensor, timing value, token, or hardware activity.
Interpretability
We try to connect internal states and computations to understandable features, mechanisms, decisions, or behavior.
A Familiar Sysadmin Analogy
A process monitor can show:
That is useful evidence. But it does not automatically tell you why the application is consuming 98% CPU.
AI interpretability faces a vastly larger version of that problem.
What Can We Potentially Inspect?
Tokens
The token IDs entering and leaving the model.
Embeddings
Learned vector representations used to begin transformer processing.
Activations
Transient internal values created as the current input moves through layers.
Attention
Queries, keys, values, scores and attention patterns, where accessible.
Weights
The trained numerical parameters that shape transformations.
Logits
Output scores over possible next tokens.
The Scale Problem
Even one model operation can involve enormous tensors. A complete inference can create huge numbers of intermediate numerical values.
Capturing everything may be expensive. Understanding everything is a different and harder problem.
Distributed Representation Makes It Harder
We should not expect every concept to live neatly in one location:
Individual units can sometimes correlate with recognizable features, but the network's useful representations are often distributed and context dependent.
Remember “Blue”
The same token can participate in very different contextual representations. Interpretability asks whether we can identify how the network represents and distinguishes those contexts internally.
Attention Gives Us Something to Look At
Attention patterns can show which earlier positions contribute to an attention operation for a given position.
That is genuine internal structure. But caution:
Other heads, MLPs, residual streams, later layers and the cumulative network state also matter.
Probe the Representation
One interpretability approach asks whether information can be detected in internal activations.
If a property is detectable, that is evidence that information about it is represented there. It does not automatically prove the model uses that information in the causal path producing the output.
Correlation Is Not Yet Causation
This is a central scientific boundary for interpretability.
Intervene on the Machine
A stronger approach is to change an internal component and observe what happens.
If controlled intervention predictably changes behavior, that can provide stronger causal evidence than observation alone.
Ablation
An ablation removes, disables, masks, or otherwise disrupts a component to study its contribution.
The exact meaning of an ablation depends on what was changed and how carefully the comparison was controlled.
Activation Patching
Another family of methods replaces selected internal activations from one run with activations from another run.
This can help investigate which internal states contribute causally to a behavioral difference. Actual methods can be considerably more sophisticated than this simplified diagram.
Features May Be More Useful Than Individual Neurons
Interpretability research often looks for directions or patterns in activation space that correspond more cleanly to useful features than single raw units do.
One modern approach uses learned decompositions such as sparse autoencoders to search for more interpretable features. These methods are research tools, not a guaranteed dictionary of everything the model “thinks.”
Mechanistic Interpretability
Mechanistic interpretability aims beyond labels such as “this activation correlates with arithmetic.” It tries to reconstruct computational mechanisms:
The ambition is closer to reverse engineering a program than merely describing model behavior from the outside.
But the “Program” Was Learned
Traditional software gives us source code written by people:
A trained neural network gives us learned matrices and architecture:
The high-level behavior is encoded in numerical structure that training discovered rather than in a human-written list of semantic rules.
Interpretability Is Reverse Engineering
Behavior
What does the model do?
Observation
What internal events can we record?
Hypothesis
What mechanism might explain those observations?
Intervention
Can changing the proposed mechanism predictably change behavior?
Do Not Ask the Model and Call That Instrumentation
We can ask a model:
It may generate a useful explanation. But that explanation is itself another model output.
Now WITNESS Becomes Obvious
WITNESS began with a demand that sounds simple:
Interpretability immediately forces a second question:
Those are different jobs.
WITNESS's Evidence Ladder
The farther we move from raw observation, the more important provenance, method, uncertainty and reproducibility become.
The Event Field Returns
Craig's earlier WITNESS observation matters here: an inference is not one simple linear timeline.
A useful forensic record may need timestamps, identities, relationships and metadata sufficient to reconstruct dependencies among events.
Observation Can Change the System
Instrumentation itself has cost.
This is why WITNESS's idea of sterile capture is difficult and important: observe enough to be useful without materially distorting the experiment being observed.
Known Gaps Matter
A trustworthy examination should be able to say:
“Unknown” is often a stronger scientific statement than filling an inaccessible region with a confident story.
Can We Tell What the Model Is Doing Inside?
We can observe many internal numerical events when instrumentation and access permit.
We can search for patterns that correlate with concepts and behaviors.
We can intervene on internal states to test causal hypotheses.
We can attempt to reconstruct learned computational mechanisms.
But observation, interpretation and causal explanation are not the same thing.
And a model's own explanation is not automatically a faithful trace of its hidden computation.
Interpretability is the attempt to cross the gap from numbers we can observe to mechanisms we can justify saying we understand.
HAAW Checkpoint
- Observability
- The ability to record or measure internal states, events, timing, or outputs.
- Interpretability
- Methods for connecting internal model states and computations to understandable features, mechanisms, or behavior.
- Probe
- An analysis used to test whether particular information is detectable in a representation.
- Correlation
- An association between an internal pattern and behavior; not by itself proof of causal influence.
- Intervention
- A controlled change to internal state or computation used to test causal hypotheses.
- Ablation
- Disabling or disrupting a component to examine its contribution.
- Activation patching
- Replacing selected internal activations between runs to study their causal role.
- Feature
- A candidate interpretable pattern or direction represented within model activations.
- Mechanistic interpretability
- Efforts to reconstruct the learned computational mechanisms that produce model behavior.
- Self-explanation
- A model-generated account of its answer; useful behavior, but not automatically direct internal instrumentation.
- Sterile capture
- WITNESS principle: collect useful evidence while minimizing disturbance to the observed system.
- Known gap
- An internal state or event that available instrumentation cannot establish; it should remain explicitly unknown.
- Central lesson
- Recording internal numbers is the beginning of investigation, not the end. Interpretation requires evidence connecting those observations to behavior and mechanism.