How AI's Actually Work — HAAW

Memory in Motion

NVMe, RAM, VRAM and the Data Path — The LLM begins as persistent data. To execute it, the machine must put the right numbers where the processors can use them, and keep moving data fast enough to feed the computation.

The Model Has to Travel

A model stored on an NVMe SSD cannot simply remain there while the GPU somehow calculates through it at full accelerator speed.

Before and during inference, model data and working state must become accessible to the processors that use them.

AI performance is therefore not only about arithmetic.

It is also about where data is stored, how much memory is available, and how quickly the required numbers can be moved to the place where computation happens.

Three Important Memory Neighborhoods

NVMe / SSD

Persistent storage. Large capacity and survives power-off. Model files can live here for months or years.

System RAM

Host working memory. Directly available to the CPU and used by the operating system, inference software, buffers, and model data.

GPU VRAM

Accelerator working memory. Very high-bandwidth local memory used for weights, activations, KV cache, and GPU working data.

The Simplified Loading Path

For a conventional discrete-GPU system, a useful first model is:

Model files on NVMe / SSD
↓
Storage I/O brings model data into the active system
↓
System RAM may stage / hold model data
↓
PCIe transfers required data toward the GPU
↓
Weights become resident in GPU VRAM
↓
GPU executes tensor operations using local data

This is a conceptual path, not a claim that every AI system performs every transfer in exactly this way. Modern systems can use memory mapping, direct-storage techniques, unified memory, accelerator modules, model sharding, and other architectures.

Physically: They Are Separate Devices

CPU / HOST PLATFORM
│
├── memory channels ─────→ SYSTEM RAM
│
├── PCIe ────────────────→ NVMe SSD
│                         └── model files
│
└── PCIe ────────────────→ GPU CARD
                         ├── GPU processor
                         └── VRAM

The NVMe SSD and GPU can both use PCI Express connectivity while remaining completely separate devices with very different purposes.

PCIe Is a Connection, Not a Storage Pool

PCI Express provides high-speed point-to-point connectivity through the platform. It allows devices and the host to exchange data.

But sharing PCIe technology does not mean NVMe storage and GPU VRAM become one physical memory device.

NVMe = persistent storage device
PCIe = interconnect / transport path
GPU card = accelerator device
VRAM = accelerator-local working memory

Once the Weights Reach VRAM

Now the physical relationship changes dramatically.

The GPU processor can repeatedly access weights and working tensors through its local memory system without sending every operation back across PCIe to the host.

GPU PROCESSOR
    ⇅  local high-bandwidth memory interface
VRAM
├── model weights
├── activations
├── KV cache
└── working buffers

Keeping frequently needed numerical data close to the accelerator is a major reason VRAM capacity and memory bandwidth matter so much.

Capacity, Bandwidth and Latency

Capacity

How much?
How many bytes of weights, caches, activations, buffers, and other data can fit.

Bandwidth

How much per second?
How rapidly large amounts of data can move through a memory or interconnect path.

Latency

How long to begin / complete an access?
The delay associated with obtaining needed data or completing communication.

These are different constraints.

A device can have enormous capacity but inadequate bandwidth, excellent bandwidth but insufficient capacity, or high throughput while individual accesses still have meaningful latency.

Why a Model Can “Fit” and Still Run Poorly

Suppose the model technically fits into available memory.

That answers only the capacity question.

If execution constantly requires moving large amounts of data across a slower boundary, waiting for memory, or shifting model portions between host and accelerator memory, throughput can suffer.

FITS IN MEMORY?  → capacity question
FED FAST ENOUGH? → bandwidth question
WAITING HOW LONG? → latency question

Why More VRAM Matters

More accelerator memory can allow more of the model and its active runtime state to remain close to the GPU.

But the memory demand is not only the saved model weights.

Weights

The trained parameters themselves can consume substantial memory.

KV Cache

Grows with active sequence/context requirements and is especially important during generation.

Activations

Intermediate tensors require working space during computation.

Runtime Buffers

Kernels and inference software need additional memory for temporary and management purposes.

When One GPU Is Not Enough

A model or serving workload may exceed the useful capacity of one accelerator.

The system can then distribute work and model state across multiple devices.

MODEL
    │
    ├── portion / work ──→ GPU 0 + VRAM 0
    ├── portion / work ──→ GPU 1 + VRAM 1
    ├── portion / work ──→ GPU 2 + VRAM 2
    └── portion / work ──→ GPU 3 + VRAM 3

Exactly how the model is divided depends on the inference framework, model architecture, accelerator topology, and chosen parallelism strategy.

Once multiple accelerators participate, communication between devices becomes another part of the data-path problem.

Offloading: Trading Speed for Reach

If all desired model state cannot remain in accelerator memory, some systems keep portions in system RAM or otherwise move data as needed.

SYSTEM RAM
    ⇅
PCIe / interconnect
    ⇅
GPU VRAM

This can make it possible to execute models that would not otherwise fit entirely in VRAM, but additional data movement can reduce performance.

So “Can I run it?” and “How fast will it run?” are separate questions.

Direct Paths and Architectural Variations

The simple NVMe → RAM → GPU path is useful for learning, but real systems can optimize it.

Some platforms support mechanisms that reduce CPU involvement or unnecessary copies when transferring data between storage, host memory, and accelerators. Integrated or unified-memory architectures can also blur some of the physical boundaries found in a traditional discrete-GPU machine.

HAAW rule:

Learn the ordinary data path first. Then treat faster or more integrated paths as architectural variations—not as contradictions of the underlying need to get numerical data to the hardware that will operate on it.

Now Follow One Request

The model is already loaded. Craig submits:

Why is the sky blue?
Prompt arrives through interface
↓
CPU / runtime prepares request and tokens
↓
Required tensors are available to accelerator
↓
GPU reads weights from VRAM
↓
GPU produces new activations
↓
Attention state contributes to KV cache
↓
Layers repeatedly read and write numerical state
↓
Logits → token selection → next decode step

During this process, the machine is not merely “calculating.” It is continually moving numerical data to and from the places where calculation can occur.

The Von Neumann Echo

There is an old computer-design lesson hiding here: processors need data, and moving data can become as important as performing arithmetic on it.

AI accelerators greatly increase computational throughput, but that makes feeding the arithmetic machinery efficiently even more important.

For large-model inference, the journey of the numbers through the memory hierarchy is therefore part of the computation story—not an implementation footnote.

The Data Path Is Part of the AI Machine

The LLM may begin as persistent numbers on storage.

Those numbers must become available in working memory. The accelerator must receive the weights and active tensors it needs. Results must be written back into memory. New results become inputs to later operations.

Storage holds. Interconnects move. Memory feeds. Processors calculate.

And throughout inference, those four jobs cooperate at extraordinary speed.

HAAW Checkpoint

NVMe / SSD
Persistent storage capable of holding model files when the system is powered off.
System RAM
CPU-accessible working memory used by the operating system, runtime, buffers, and potentially model data.
VRAM
High-bandwidth accelerator-local memory used for model weights and active computational state.
PCIe
A high-speed host/device interconnect; it does not make NVMe and VRAM one shared physical device.
Capacity
The amount of data a memory can hold.
Bandwidth
The rate at which data can be transferred.
Latency
The delay associated with accessing or communicating data.
Offloading
Keeping some model state outside accelerator memory and transferring it when required, often trading performance for the ability to run a larger workload.
Data path
The physical and logical route numerical information follows among storage, memory, interconnects, and processors.