Why Add Another GPU?
There are several different reasons to use multiple GPUs, and they should not be confused.
The Model Is Too Large
Its weights and required runtime state cannot fit comfortably into one GPU's VRAM.
One GPU Is Too Slow
The computation can sometimes be divided so several accelerators contribute to processing one workload.
Too Many Requests
Multiple GPUs can increase serving capacity by handling more users, batches, or model replicas.
Each accelerator has its own processor and usually its own local memory. Software must decide how work and data are distributed.
Start With Two Physical Cards
CPU + RAM + NVMe
│
├── PCIe / interconnect ──→ GPU 0 ⇄ VRAM 0
│
└── PCIe / interconnect ──→ GPU 1 ⇄ VRAM 1
VRAM 0 and VRAM 1 are normally distinct physical memory pools. Installing two 24 GB cards does not, by itself, create one ordinary 48 GB block of VRAM that every operation can access identically.
Software and hardware interconnects must cooperate to use the two devices as part of one model execution.
The First Problem: Where Do the Weights Go?
Suppose the model's working requirements exceed the practical memory available on one GPU.
One approach is to divide the model state across accelerators.
│
├── portion of model ──→ GPU 0 / VRAM 0
│
└── portion of model ──→ GPU 1 / VRAM 1
This general idea is called model parallelism. But there are several ways to divide the model.
Strategy 1: Pipeline Parallelism
One conceptually simple division is to place different groups of transformer layers on different accelerators.
│
▼
GPU 0
Layers 1 → 2 → 3 → ... → N
│
│ activation transfer
▼
GPU 1
Layers N+1 → ... → final layer
│
▼
logits
GPU 0 performs its portion, then intermediate activations must reach GPU 1 so processing can continue.
the output of one accelerator becomes input to another. Communication is now part of the critical computation path.
Strategy 2: Tensor Parallelism
Instead of assigning whole groups of layers to different GPUs, a single large tensor operation can itself be divided across accelerators.
│
┌───────────┴───────────┐
▼ ▼
GPU 0 GPU 1
part of operation part of operation
│ │
└───────────┬───────────┘
▼
combine / exchange partial results
▼
complete result
Now both accelerators may participate in the same layer's computation. This can expose substantial parallelism, but it can also require frequent high-speed communication among devices.
Strategy 3: Replication / Data Parallel Serving
Sometimes we do not need to split one model execution at all.
If the model fits on each GPU, multiple GPUs can hold separate model replicas and process different requests or batches.
Request B ──→ GPU 1 [full model replica]
Request C ──→ GPU 2 [full model replica]
Request D ──→ GPU 3 [full model replica]
This increases serving throughput, but it does not combine the VRAM of those replicas to make a single oversized model fit.
Three Ideas — Do Not Mix Them Up
Pipeline Parallel
Different groups of layers or stages execute on different devices.
Tensor Parallel
Individual tensor operations are partitioned so multiple devices cooperate within a layer.
Replica / Data Parallel
Multiple copies of a model process separate work, increasing aggregate throughput.
Large production systems can combine these techniques rather than choosing only one.
The New Cost: Communication
On one GPU, much of the heavy traffic can remain between the GPU processor and its local VRAM.
With multiple GPUs cooperating on one inference, some information must cross device boundaries.
VRAM 0 VRAM 1
Depending on the hardware, topology, and platform, communication may use PCIe or specialized accelerator interconnects.
The faster the GPUs calculate, the more painful it can become if one accelerator repeatedly waits for another device's data.
Topology Matters
It is not enough to count GPUs. We also need to know how they are connected.
GPU 0 ⇄ fast direct link ⇄ GPU 1
Example B:
GPU 0 ⇄ PCIe fabric / switches ⇄ GPU 1
Example C:
GPU 0 ⇄ host / network fabric ⇄ GPU in another server
Those systems may contain the same nominal number of accelerators while having very different communication characteristics.
GPU model, VRAM, memory bandwidth, PCIe generation and lane arrangement, peer connectivity, topology, software, and workload all matter.
Now Our Data Path Has Branches
On the previous page we followed:
With several accelerators, we may instead have:
│
▼
system RAM / runtime
│
├────────────→ GPU 0 / VRAM 0
├────────────→ GPU 1 / VRAM 1
├────────────→ GPU 2 / VRAM 2
└────────────→ GPU 3 / VRAM 3
and possibly:
GPU 0 ⇄ GPU 1 ⇄ GPU 2 ⇄ GPU 3
The inference runtime must understand where model pieces live and coordinate computation and communication among them.
Follow One Token Across Two GPUs
Use a simplified pipeline example.
During autoregressive generation, some form of this coordinated work occurs again for the next decode step.
The Slowest Link Can Matter
Suppose GPU 0 finishes its work but GPU 1 cannot proceed until required data arrives.
GPU 1 waits.
If this happens repeatedly, additional accelerator arithmetic capability does not automatically translate into proportional end-to-end speed.
↓
WAIT FOR DATA
↓
fast computation
↓
WAIT FOR DATA
This is why multi-GPU engineering is partly a problem of communication, synchronization, and workload placement.
Two 24 GB GPUs: What Does “48 GB” Mean?
Physically, two 24 GB discrete cards provide 48 GB of aggregate VRAM capacity across the two devices.
But that does not mean every program sees one transparent, uniform 48 GB memory space with the same access characteristics as one hypothetical 48 GB GPU.
GPU 1 → 24 GB local VRAM
-----------------------------
48 GB aggregate physical capacity
≠ automatically one uniform 48 GB local-memory pool
To use the aggregate capacity for one model, the software must partition model state and coordinate the required cross-device computation.
What the Runtime Now Has to Know
Placement
Which weights, layers, caches, and buffers belong on which device?
Scheduling
Which accelerator performs which operation, and when?
Communication
Which tensors or partial results must cross device boundaries?
Synchronization
Which computation must wait until another device has completed its work?
The model has not magically become aware that several GPUs exist. The surrounding execution system coordinates the distributed hardware.
More GPUs Change the Shape of the Machine
One GPU let us imagine the model weights and computation living close together.
Multiple GPUs give us more aggregate memory and computation, but they introduce boundaries.
Across every boundary, data may have to move. Whenever cooperating computations depend on that data, communication can become part of the critical path.
So scaling an AI system is not merely adding arithmetic engines. It is designing a machine in which the arithmetic engines can work together efficiently.
HAAW Checkpoint
- Multi-GPU system
- A computer or distributed system in which more than one accelerator participates in AI workloads.
- Model parallelism
- Dividing one model's state or computation across multiple devices.
- Pipeline parallelism
- Assigning different stages or groups of model layers to different accelerators.
- Tensor parallelism
- Splitting individual tensor operations across multiple accelerators that exchange or combine partial results.
- Replication / data parallel serving
- Keeping model replicas on multiple accelerators so separate requests or batches can be processed concurrently.
- Aggregate VRAM
- The total physical accelerator-memory capacity across devices; it is not automatically one uniform local-memory pool.
- Interconnect
- The communication path through which devices exchange data, such as PCIe or specialized accelerator links.
- Topology
- The physical and logical arrangement of device connections; it affects how efficiently data can move among accelerators.
- Scaling lesson
- Adding GPUs adds computation and memory, but also introduces communication, placement, and synchronization problems.