Where We Left the Machine
The computer is running, but the model is still sitting on storage.
Linux has booted. Filesystems are mounted. The CPU and system RAM are active. The GPU or other accelerator has been discovered, its driver is available, and the machine's supporting services are running.
Now an AI runtime or inference service starts.
The model is about to become ready for computation, but it has not received our question and it is not generating tokens.
What Is the “Model” on Disk?
A trained model is not stored as sentences, answers, memories, or a giant database of conversations.
At a simplified level, the files needed to run a language model contain several kinds of information:
Model Architecture / Configuration
Information describing the model's structure: dimensions, number of layers, attention arrangement, vocabulary-related settings, and other parameters needed to construct the computational graph.
Weights
Very large collections of numerical values produced by training. These values determine how signals are transformed as computation passes through the model.
Tokenizer Information
Rules and vocabulary data used to convert incoming text into tokens the model can process, and tokens back into text we can read.
Supporting Metadata
File format, tensor names and shapes, precision or quantization information, and other data needed by the runtime to interpret the stored model correctly.
The Weights Are Numbers
This is one of the most important ideas in HAAW.
The learned portion of the model is represented largely by enormous arrays of numbers.
Those numbers are organized into tensors—multidimensional arrays used by the mathematical operations of the neural network.
A model described as having billions of parameters contains billions of learned numerical parameter values. The exact representation may vary: for example, weights may be stored using different numerical precisions or may be quantized to use fewer bits.
The numbers by themselves do nothing while they remain on the NVMe drive. They become operational only when software uses them in the model's computations.
1. The AI Runtime Starts
The inference software begins as an ordinary process or collection of processes launched by the operating system or a service manager.
Depending on the implementation, this software may include a model-serving application plus libraries that know how to perform tensor operations on the available accelerator.
└── AI inference service
├── model configuration
├── tokenizer
├── tensor / accelerator runtime
└── model weights
2. The Runtime Examines the Model
The software must know what it is loading. It reads configuration and model metadata so it can establish the required structures and associate stored weight tensors with the computations that will use them.
This is somewhat like loading any complex executable data structure: addresses, shapes, types, sizes, and relationships matter.
A weight tensor intended for one layer cannot simply be substituted blindly for a differently shaped tensor elsewhere.
3. Memory Is Allocated
The runtime requests memory needed to operate the model.
Some information remains in normal system RAM. Accelerator computations require data to be accessible to the GPU or other accelerator, so memory is allocated there as well.
persistent model files
data at rest
model data available to running software
data prepared for computation
The exact route is implementation-dependent. Data may pass through system RAM before reaching accelerator memory, may be memory-mapped, streamed, sharded, or loaded using mechanisms optimized for the particular hardware and runtime.
For HAAW, the essential idea is simpler: the model's numerical data must become accessible to the hardware that will perform its calculations.
4. Weight Tensors Move Into Working Memory
The stored weight data is read from persistent storage and made available to the runtime. For GPU execution, much or all of the required weight data is commonly placed in GPU-accessible memory such as VRAM.
For a model too large for one accelerator, this picture becomes more complicated. Weights may be divided across multiple GPUs or even multiple machines. We will deal with distributed models later; our reference machine still uses the simplest useful case.
5. The Runtime Prepares Computation
Loading weights is not necessarily the final startup step. The runtime may create internal data structures, reserve temporary working memory, initialize accelerator kernels, prepare caches, or optimize the computational representation for the installed hardware.
Some systems may also perform a small warm-up computation before accepting normal requests.
A loaded model is better understood as a very large mathematical system whose learned numerical parameters are now available for use.
It is waiting for input.
Where Did the Model's Knowledge Go?
This question deserves care.
When the weight tensors are loaded, the learned numerical parameters produced during training are now available to the running model. What we casually call the model's “knowledge” is not copied into a separate collection of readable facts in VRAM.
Instead, learned relationships are encoded across the model's parameters and structure. Their effect becomes visible when those parameters participate in computation.
We will return to this repeatedly. For now, the safe mechanical statement is:
What Is in VRAM Now?
At a simplified level, VRAM may now contain:
Model Weights
The learned numerical tensors used repeatedly during inference.
Runtime Structures
Memory and data structures needed by the accelerator software to execute model operations.
Working Space
Memory reserved for intermediate calculations that will appear once inference begins.
Caches
Depending on the runtime and state of execution, memory may be reserved or later used for caches such as the transformer KV cache.
Not every implementation arranges memory identically. HAAW will distinguish this general model from implementation-specific details when they matter.
The Machine Is Ready
We began with a powered-off computer.
Linux booted. The accelerator became usable. The inference runtime started. The trained model was read from storage. Its numerical parameters were made available to the hardware that will perform the model's calculations.
But the model has still not answered anything.
No user prompt has entered our journey.
That changes next.
Our Journey So Far
HAAW Checkpoint
- Established mechanism
- A trained model contains learned numerical parameters organized as tensors and supporting information required to run the model.
- Established mechanism
- The runtime makes those parameters accessible to the processor or accelerator performing inference.
- Important distinction
- Stored weights, loaded weights, and active inference are different states.
- Interpretive caution
- Calling weights “knowledge” can be useful shorthand, but it must not imply that VRAM contains a conventional readable database of facts.
- Still ahead
- The first user input: text enters the system and must somehow become something this mathematical machine can process.