The memory wall

The gap that defines modern optimization

A 4 GHz core retires a simple instruction in about 0.25 ns. A fetch from main memory costs 60–100 ns. That is a factor of 240–400: while one load waits on DRAM, the core could have executed a few hundred instructions — and instead it executes none.

This gap was not always there. In 1990, CPU and memory speeds were within an order of magnitude of each other. Since then, compute throughput has grown far faster than memory latency has shrunk, and the divergence compounded for three decades. The industry’s answer was not to close the gap — physics wouldn’t allow it — but to hide it, with a hierarchy of caches: L1 at roughly 1 ns, L2 around 3–4 ns, L3 around 10–12 ns, and DRAM at the bottom.

Everything in this course descends from one consequence: the cost of your code is dominated by where its data lives, not by what its instructions do. An algorithm that touches memory in the pattern the hierarchy was built for runs hundreds of times faster than the same algorithm fighting it. That multiplier is the budget DOTS spends.

The 64-byte contract

Caches don’t move individual bytes. The unit of transfer between memory and cache is a cache line: 64 bytes, aligned. Ask for one byte and the hardware fetches the 64-byte line containing it — always. There is no smaller transaction.

That gives you a lever and a tax. The lever: bytes adjacent to the one you asked for are now free — already resident, ~1 ns away. The tax: any of those 64 bytes you don’t use were still paid for, in bandwidth and in cache capacity that could have held something useful.

Consider a typical gameplay object — position, rotation, health, a few references, some flags — easily 100+ bytes per instance. Your loop reads one float of it:

Four useful bytes per 64 fetched is a 16× bandwidth tax. Run that loop over ten thousand objects and you haven’t written a slow algorithm — you’ve written a fast algorithm that spends 94% of the memory system’s effort hauling bytes nobody reads.

Now invert the layout. Put every instance’s Health value contiguous in its own array. The same loop reads a line and gets sixteen useful floats: the next fifteen iterations are already resident before they’re requested. Nothing about the algorithm changed. Only the layout did — and the layout decided the cost. This is the difference between array-of-structures (AoS: whole objects side by side) and structure-of-arrays (SoA: each field in its own array), and it is why SoA recurs through every layer of DOTS.

The prefetcher’s bargain

Caches hide latency for data you’ve already touched. The second mechanism, the hardware prefetcher, hides it for data you’re about to touch — under one condition.

The prefetcher watches the stream of cache misses for patterns. Sequential addresses, or a constant stride, are patterns it recognizes; it then fetches lines ahead of your loop, so by the time iteration i + 4 executes, its line has been in flight for hundreds of cycles and arrives on time. A linear walk over a big array can approach the speed of pure L1 access not because the data was cached, but because every miss was started early. The bargain: make your next address a function of your current one, and latency disappears.

Pointer-chasing breaks the bargain — not because the prefetcher is bad at guessing, but because there is nothing to guess from. Walk a linked list, or an array of references to scattered heap objects, and the address of node n + 1 is stored inside node n. It cannot even be computed until node n’s load completes. Every step is a full, unhidden, serialized memory latency: a 10,000-node walk is 10,000 × ~80 ns of the core doing nothing, by construction. No cache size, clock speed, or compiler flag changes this — the dependency chain is in the data structure itself.

This is why “just use faster hardware” fails as an optimization strategy, and why an object graph of scattered heap allocations — the default shape of idiomatic object-oriented gameplay code — has a performance ceiling no amount of instruction-level cleverness can lift. The fix has to change where the data lives, which is precisely the part classic object-oriented design treats as an invisible implementation detail.

Where the mechanism breaks down

The story so far makes contiguity sound like a universal answer. It isn’t, and the failure cases matter:

• Genuinely random access defeats everything. If your access pattern is data-dependent and unpredictable — hash lookups, spatial queries into scattered cells — contiguous storage doesn’t help, because you won’t touch neighbors. The optimization then moves up a level: sort the work so access becomes sequential, or accept the misses and batch them so they overlap.
• Bandwidth is a shared, finite pipe. Perfect streaming from every core at once will saturate memory bandwidth before it saturates compute. At that point more threads make nothing faster. You’ll measure this yourself in the Jobs module.
• Writes are not free reads. A store to a line you don’t own forces ownership traffic between cores. Two threads writing to the same line — even different bytes of it — ping-pong that line between them. This is false sharing, and it turns “embarrassingly parallel” loops into serialized ones. It gets a full lesson later.

The memory wall is not an argument that contiguity is always right. It is the cost model that lets you predict when it’s right — which is what the rest of the course trains.

What this buys you

Hold onto the three numbers — ~0.25 ns per instruction, ~64 bytes per line, ~60–100 ns per miss — and most of the architecture you’re about to learn stops being design taste and becomes arithmetic. Why entity data lives in fixed-size contiguous blocks, why components are plain data grouped by type, why iteration order is dictated by the runtime rather than by you: each is the layout that makes loops walk memory linearly, with the prefetcher’s bargain intact. You could derive the shape of the solution from this lesson alone. The next module shows the one Unity actually built — and measures whether the arithmetic holds on your machine.