Cache + TLB

Modern CPUs are fast enough that direct RAM access would constantly slow them down. To reduce this bottleneck, CPUs use multiple layers of acceleration.

Two of the most important are:

TLB → speeds up address translation
CPU Cache → speeds up actual data access

The TLB helps the CPU determine where memory is located. CPU cache helps the CPU retrieve the memory contents quickly.

The Full Memory Path

When a program accesses memory, several things happen internally.

Program uses virtual address ↓ TLB lookup ↓ Virtual → physical translation ↓ CPU cache lookup ↓ Data returned to CPU

All of this may happen billions of times per second.

Step 1: Address Translation

Programs use virtual addresses, not physical RAM addresses.

Before memory can be accessed, the CPU must translate the virtual address into a physical address.

Virtual Address ↓ TLB ↓ Physical Address

The TLB caches recent translations to avoid repeated page-table walks.

Fast Path

TLB hit ↓ Immediate translation

Slow Path

TLB miss ↓ Walk page tables ↓ Load translation into TLB

Even before data is fetched, the CPU must first determine where the data lives.

Step 2: CPU Cache Lookup

After translation, the CPU checks cache memory for the requested data.

Physical Address ↓ L1 Cache ↓ L2 Cache ↓ L3 Cache ↓ RAM

CPU caches are much faster than RAM.

Cache Levels

L1 Cache → smallest and fastest
L2 Cache → larger and slower
L3 Cache → shared and larger still

Cache Hit vs Cache Miss

Cache Hit

Data found in cache ↓ Fast access

This is the ideal case.

Cache Miss

Data not in cache ↓ Fetch from lower cache or RAM ↓ Higher latency

RAM access is dramatically slower than cache access. Modern CPUs rely heavily on cache efficiency.

Combined Behavior

Good performance often depends on both:

high TLB hit rates
high cache hit rates

Good locality ↓ TLB reuse ↓ Cache reuse ↓ High performance

Locality

Programs often reuse nearby memory repeatedly. This behavior is called locality.

Temporal Locality

Recently used data is likely to be used again soon.

Spatial Locality

Nearby memory is likely to be accessed together.

Locality is one reason caches and TLBs work so well. Programs naturally tend to reuse nearby memory.

What Hurts Performance

Some memory access patterns reduce TLB and cache effectiveness.

Common Problems

Large random memory access
Poor locality
Large sparse data structures
Frequent context switching
Huge working sets

A program can have enough RAM and still perform poorly because of bad cache or TLB behavior.

Latency Differences

Approximate access costs:

L1 Cache → very fast L2 Cache → fast L3 Cache → slower RAM → much slower Disk/Swap → extremely slow

The further the CPU must travel for data, the longer execution stalls.

Why Modern CPUs Are Complex

Modern CPUs spend enormous effort trying to avoid waiting for memory.

This is one reason processors contain:

multiple cache levels
large TLB structures
prefetchers
out-of-order execution
branch prediction

A significant part of modern CPU design is really about hiding memory latency.

Observing Cache and TLB Behavior

perf stat

perf stat <command>

Possible Events

perf stat -e cache-misses,cache-references <command> perf stat -e dTLB-loads,dTLB-load-misses <command>

List Available Events

perf list

Available counters depend on the CPU architecture and Linux kernel support.

Conceptual Summary

The TLB accelerates address translation. CPU caches accelerate data retrieval.

Together they allow modern CPUs to access memory far faster than raw RAM latency would otherwise allow.

Much of modern performance depends on keeping both the TLB and caches efficient.

Visual Model

Follow how the CPU searches through cache levels before accessing main memory.