AI Memory
Weights, Context, KV Cache, Agent State and Persistent Storage Are Not the Same Thing
“Memory” Is Doing Too Much Work
People naturally say an AI remembers something. Technically, that one word can describe several very different mechanisms.
An AI system may appear to remember because information is encoded in trained weights, because it is present in the current context, because numerical attention state remains in a KV cache, because an agent carries task state forward, or because information was saved in persistent external storage and retrieved later.
Before asking, “Does the AI remember?” ask: Which kind of memory are we talking about?
The Five Boxes
Weights
Long-lived numerical parameters produced by training. They shape model behavior but are not a conventional searchable notebook of facts.
Context
Information represented for the current inference: instructions, conversation, retrieved material, tool results and the current request.
KV Cache
Temporary numerical attention state derived while processing the current sequence. It helps generation continue efficiently.
Agent State
The working record of a multi-step job: goal, constraints, observations, tool results, completed steps, errors and remaining work.
Persistent Storage
Information kept outside the core model—files, databases or memory services—that can later be retrieved and supplied again.
1. Weights — What Training Left Behind
Training repeatedly adjusted enormous arrays of parameter values. When training ended, those values were saved as model weights.
During ordinary inference, the model uses those weights. It does not normally rewrite them because a user asks a question or supplies a new fact. There is usually no single address labeled “this fact.” Learned information is distributed through parameter relationships.
2. Context — What the Model Has for This Inference
Suppose Craig says, “My test machine has two RTX 3090 GPUs.” If that statement is included in usable context, it can affect the answer immediately. No retraining is required.
Context may include system or service instructions, application instructions, selected earlier conversation, retrieved documents, tool results and the current message. Context is not weights. Making a fact available now does not ordinarily make it a permanent learned parameter.
3. KV Cache — Numerical State, Not a Little Transcript
During transformer attention, key and value representations are produced. During generation, retaining useful K/V state from already-processed sequence positions avoids repeating substantial work for every new token.
The KV cache is temporary computational memory. It is not a text file containing the conversation; it consists of numerical tensors used by later attention computation. Its organization varies by architecture and serving implementation, and its size can become an important memory cost as context grows.
4. Agent State — “Where Are We in the Job?”
Page 31 gave us an agent diagnosing a failed HAAW web server. After several operations its task state might contain:
Observation 1: Host reachable.
Observation 2: httpd failed.
Observation 3: Log reports a configuration syntax error.
Verification: Configuration test independently reports the same failure.
Status: Enough evidence to report; no repair authorized.
That is agent state. Some may be placed in model context. Some may live in an orchestrator, database, file or other external structure, with only selected portions supplied to the model.
Agent state answers: “Where are we in this job?”
5. Persistent Storage — Survive, Retrieve, Reuse
A system can deliberately store information outside the model and retrieve it during a later request.
The store might be a database, document, key-value store, vector index, ordinary file or specialized memory service. Retrieval makes selected stored information available to later inference; it does not require the transformer to absorb it permanently into trained weights.
Now Add the Physical Machine
Here two different uses of memory collide. Weights, context, KV cache and agent state describe roles in the AI system. RAM, VRAM and NVMe describe physical places where bits and numbers may reside.
“What kind of memory is this?” and “Where is it physically stored?” are two different questions.
| Logical/System State | Possible Physical Location | Typical Lifetime |
|---|---|---|
| Model weights | NVMe/SSD at rest; RAM and/or VRAM while loaded | Long-lived model/checkpoint |
| Context source material | Files/databases/RAM; model-working forms may reach accelerator memory | Request/session/system dependent |
| KV cache | Commonly VRAM; implementations can offload or distribute it | Sequence/request lifetime |
| Agent state | Application RAM, database, files or orchestrator storage; selected state may enter context | Task dependent |
| Persistent storage | SSD/NVMe, database service, remote durable storage | Designed to survive requests and possibly restarts |
Exact placement varies by architecture, runtime, deployment and optimization strategy.
The Power-Off Test
NVMe / Durable Storage
Stored model files and deliberately persisted information can remain after power is removed.
RAM
Ordinary volatile system memory loses its working contents when power is removed.
VRAM
Loaded working state such as tensors and KV cache is not persistent storage after shutdown.
After restart, software can reload weights and retrieve durable application information. Previous transient activations and KV cache are not magically reconstructed unless the system separately stored enough information to recreate or resume them.
“Do You Remember Our Earlier Conversation?”
The model may use it directly.
The surrounding system may retrieve it into new context.
Training may have influenced weights—a different mechanism.
The system may simply be carrying task progress across inferences.
The visible behavior—“it remembered”—does not by itself tell us which mechanism produced it.
The Human-Memory Metaphor Has Limits
Human biological memory is not equivalent to model weights, context windows, KV tensors or databases. The word is useful shorthand, but it can hide architecture if the metaphor becomes the explanation.
The WITNESS Question
If an AI system uses information during an experiment, saying merely that “the AI remembered it” is not enough. WITNESS should ask whether it came from weights, request context, retrieval, a previous tool result or agent state; what was actually supplied to this inference; where and when the state was created or modified; and what provenance connects it to its source.
That converts “the AI remembered” from an interpretation into architectural and evidentiary questions.
The Boundary
Weights are durable learned model parameters.
Context is information available to the current inference.
KV cache is temporary numerical attention state.
Agent state records where a multi-step task stands.
Persistent storage keeps information outside the core model for possible later retrieval.
RAM, VRAM and NVMe are physical memory/storage technologies—not synonyms for those logical roles.
One word—memory—can hide several entirely different mechanisms.
HAAW Checkpoint
Weights: Learned numerical parameters produced by training and used during inference.
Context: Information represented for the model during a particular inference.
Context Window: Finite token capacity available under the architecture/configuration/service.
KV Cache: Temporary cached key/value attention state derived from processed sequence positions.
Agent State: Task-level information maintained across inference/action cycles.
Persistent Storage: Durable external storage from which information can later be retrieved.
RAM: Volatile host working memory.
VRAM: Accelerator-local working memory commonly used for weights, activations, KV cache and runtime buffers.
NVMe/SSD: Persistent storage commonly used for model files, data and durable application state.
Retrieval: Bringing selected external information into a request or task so it can affect later inference.
Central lesson: First identify the role of the information. Then ask where that information physically lives.
Permissions and Control — What Is the AI Allowed to Read, Write, Change and Execute?