The GPU Has a Job
The GPU does not contain the intelligence of the LLM. Its job is to execute huge quantities of the mathematical operations required by the model—very quickly.
Originally developed around highly parallel graphics workloads, GPUs are also extremely useful for workloads in which large arrays of numbers undergo many similar operations.
That description fits modern neural-network computation remarkably well.
The model supplies learned numbers and mathematical structure. The GPU performs much of the arithmetic needed to execute that structure.
The GPU Card Is More Than the GPU Chip
When we say “GPU,” we sometimes mean the processor itself and sometimes the entire accelerator card. Physically, a discrete GPU card contains several important systems.
│
├── GPU processor — parallel computation
├── VRAM — high-bandwidth accelerator memory
├── memory controllers
├── PCIe interface — connection to host system
├── power-delivery circuitry
└── cooling — heatsink / fans or other cooling system
CPU Versus GPU: Different Strengths
CPU
A relatively small number of sophisticated general-purpose processing cores, excellent for operating-system work, control flow, applications, and many varied tasks.
GPU
A highly parallel processor designed to perform large numbers of suitable operations concurrently, making it effective for the tensor computations used by neural networks.
This is not “CPU slow, GPU fast.” A CPU and GPU are optimized for different kinds of work. AI systems use both.
What Is Actually Being Calculated?
Remember our earlier transformer page:
K = X × WK
V = X × WV
Those compact equations hide enormous collections of multiply-and-add operations.
The model repeatedly transforms arrays of numerical values using learned weight arrays. Attention projections, feed-forward networks, output projections, and many other operations rely heavily on matrix and tensor arithmetic.
The symbols on our transformer diagrams become real instructions moving real numerical data through physical processor hardware.
From Matrix to Tensor
Scalar
A single numerical value.
Vector
An ordered one-dimensional collection of values.
Matrix
A two-dimensional arrangement of values—rows and columns.
Tensor
A general multidimensional array of numerical values. In AI software, “tensor” is the common container for model data and intermediate results.
Weights, embeddings, activations, attention data, and logits can all be represented using tensors of different shapes.
Parallelism: Do Lots of Arithmetic at Once
Suppose a computation requires the same kind of arithmetic across a huge number of tensor elements.
Instead of asking one general-purpose execution path to perform every operation serially, a GPU can schedule suitable work across many parallel execution resources.
│
▼
work divided into many smaller operations
│ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼
parallel GPU execution resources
│ │ │ │ │ │
└────┴────┴────┴────┴────┘
▼
result tensor
What Are “GPU Cores”?
The word core can be misleading because a GPU “core” is not simply a tiny copy of a CPU core.
Modern GPUs organize arithmetic resources into larger processing groups. NVIDIA, for example, groups execution resources into Streaming Multiprocessors (SMs); other vendors use their own architectures and terminology.
Inside these structures are arithmetic units capable of carrying out many concurrent operations.
the accelerator contains many execution resources arranged to process suitable numerical workloads in parallel.
Specialized Matrix Hardware
Modern AI accelerators often include hardware specialized for matrix operations at numerical formats commonly used in machine learning.
On NVIDIA GPUs these are commonly called Tensor Cores. Other accelerator designs provide comparable specialized matrix-compute capabilities under different names.
These units can greatly accelerate the dense mathematical operations that appear repeatedly in transformer inference.
VRAM: Keep the Numbers Close
Fast arithmetic hardware is of little use if it spends most of its time waiting for data.
The GPU therefore has access to high-bandwidth memory—on a discrete GPU card, typically VRAM—designed to feed its computational machinery rapidly.
Capacity and Bandwidth Are Different
VRAM Capacity
How much data the accelerator memory can hold. Large models, long contexts, KV cache, activations, and serving workloads compete for this space.
Memory Bandwidth
How quickly data can be transferred between accelerator memory and the GPU's computation machinery.
A GPU can therefore have enough memory to hold a workload yet still be limited by how rapidly data can be supplied to the arithmetic units—or vice versa.
PCIe and VRAM Are Not the Same Highway
This is an important physical distinction.
CPU + system RAM + NVMe
│
PCIe
│
▼
GPU CARD
GPU processor ⇄ VRAM
↑
high-bandwidth local memory path
PCIe connects the accelerator to the host system. The GPU's local memory system connects its processor to VRAM at much higher local bandwidth.
This is one reason moving data and keeping frequently needed model data near the GPU matter so much.
Return to “Why Is the Sky Blue?”
Our little question looked harmless.
But once tokenized and represented numerically, it triggered layer after layer of tensor operations:
And during generation, that cycle repeated token after token.
The speed is remarkable because enormous amounts of arithmetic are being performed while the human experiences only a short pause and then flowing text.
What the GPU Does Not Do by Itself
It Does Not Supply the Model
The trained weights must come from the model data loaded by the software system.
It Does Not Define the Prompt
The request arrives through the surrounding application and inference system.
It Does Not Invent the Architecture
The model architecture determines which mathematical operations need to be executed.
It Does Not Equal the AI System
It is one powerful physical component inside a larger hardware-and-software stack.
The Math Has a Physical Home
We can now connect our abstract transformer diagrams to hardware.
Tensors are numerical data in memory. Model weights are numerical data. Activations are numerical data. The GPU repeatedly moves, multiplies, accumulates, transforms, and writes numerical data as the model executes.
The astonishing part is not that the machine escaped mathematics.
The astonishing part is how much mathematics it can perform, in parallel, quickly enough for a conversation to feel immediate.
HAAW Checkpoint
- GPU
- A highly parallel processor used to execute large portions of neural-network computation.
- GPU card
- A physical accelerator assembly containing the GPU processor, VRAM, memory interfaces, PCIe connectivity, power delivery, and cooling.
- Tensor
- A multidimensional numerical array used to represent model data and intermediate computational state.
- VRAM
- High-bandwidth accelerator memory used to keep weights and working data close to GPU computation.
- Capacity
- How much accelerator-memory data can be held.
- Bandwidth
- How rapidly data can move between memory and computation resources.
- PCIe
- The host connection used by discrete accelerators; it is distinct from the GPU's local high-bandwidth connection to VRAM.
- Boundary
- The GPU executes the model's mathematics. It is not itself the trained LLM.