GPU clusters, TPUs, and why AI needs specialized compute.
A CPU is brilliant at sequential work. It has a handful of powerful cores — modern servers might have 64 or 128 — each capable of complex logic, branching, and decision-making. That's exactly what you need for an operating system, a database, or a web server.
AI training is a different problem. Training a model means doing the same mathematical operation — matrix multiplication — billions of times, in parallel, across enormous arrays of numbers. A CPU with 128 cores is a specialist surgeon. What AI needs is an army of thousands doing simple repetitive work simultaneously.
A Graphics Processing Unit was originally designed to render video game frames — thousands of pixels, all needing color calculations at the same time. That parallelism turns out to be exactly what AI training needs.
A modern AI GPU — NVIDIA's H100, for example — contains over 16,000 CUDA cores. Each core is simple. Together they can perform the matrix math that underlies AI training at a scale no CPU can match. Training a large language model requires thousands of these GPUs running in parallel for weeks or months.
No single GPU — no matter how powerful — can train a frontier AI model alone. Training runs on clusters: thousands of GPUs connected by high-speed interconnects 📖, coordinated to work on different parts of the problem simultaneously.
NVIDIA's NVLink and InfiniBand interconnects move data between GPUs at hundreds of gigabytes per second. The cluster is essentially a distributed supercomputer where the network speed between nodes matters almost as much as the GPU speed itself. A bottleneck in the interconnect slows the entire training run.
This is infrastructure you would recognize — racks, cooling, power distribution, redundant networking — but at a scale that requires purpose-built data centers with power draw measured in megawatts.
Two memory constraints shape AI hardware design:
GPU VRAM 📖 — the memory on the GPU itself. This is where the model weights and the current batch of training data must fit during a training step. An H100 has 80GB of VRAM. A model with hundreds of billions of parameters cannot fit in a single GPU's VRAM — it must be sharded across many GPUs. This is called model parallelism.
Training data storage — the raw dataset used for training can be petabytes in size. High-speed parallel storage systems (think Lustre or GPFS at HPC scale) feed data to the GPUs fast enough to keep them busy. A GPU sitting idle waiting for data is wasted compute — and at $30,000+ per GPU, idle is expensive.
Training hardware and inference hardware are not the same. Training needs maximum parallel throughput — thousands of GPUs grinding through data for months. Inference needs low latency — a response delivered in milliseconds to a user who just hit Enter.
Inference can run on smaller, cheaper hardware than training. A single high-end GPU can serve many simultaneous inference requests. Companies like Anthropic and OpenAI run separate inference clusters — optimized for speed and availability — distinct from their training infrastructure.
You can also run smaller models locally. Tools like Ollama (covered in the CLI Tools series) let you run a 7B or 13B parameter model on a laptop GPU or even CPU-only — the model is small enough to fit, and inference is fast enough to be useful.
For most people using AI — including BU ITS staff — the hardware layer is invisible. You interact with an API endpoint. Somewhere behind it, inference clusters handle your request. You never see the GPUs.
But understanding the hardware explains several things you will notice: