The first copy with no obligation
Why is a cache's exchange rate the worst in the system?
The first break: a copy with no obligation
Every system you have studied before a cache works with one authoritative copy of the data. A database buffer pool (the database's own memory area for disk pages) holds the copy that is in use now. A write-ahead log makes that copy survive a crash. Replication, in a database, makes several copies and uses a protocol to keep them agreeing.
A cache is different. It is the first thing that deliberately creates a second copy with no protocol obliging it to be correct, and then reads that copy in preference to the truth. That single decision generates everything here.
It gives us a precise definition:
This sentence draws a clear line. It is not a matter of degree.
- A database buffer pool page cannot be stale. Every reader and every writer goes through the buffer pool. The only possible disagreement with the disk block is that the cache is newer, and the write-ahead log (WAL, the append-only record of changes written before the data) already handles that case.
- A CPU cache cannot be stale. A hardware protocol (MESI) makes the processor cores coordinate, and that coordination removes any bypass path.
- A Redis entry in front of Postgres, a result cache, a metadata cache and a materialized view (a stored, precomputed query result) can all be bypassed. Anyone can update Postgres directly. So they are all stale by construction.
A common picture of the memory hierarchy is: CPU cache, then buffer pool, then local SSD, then distributed cache, as if this were one smooth line. In fact the line crosses the boundary above between the second and the third item. The first two are mandatory and coherent. The last two are optional and bypassable. The picture hides the most important distinction of the subject.
The second break: the benefit belongs to the workload
For most optimizations, the speed-up is a property of the mechanism. Compression makes files smaller. Vectorization (processing many values per CPU instruction) makes loops faster. Better join algorithms do less work. You can measure the gain by testing the mechanism on any data.
A cache is different. Its payoff is a property of the workload. The hit rate lives in the access pattern, not in your code. This is the only part of data engineering where the honest answer to "should we cache this?" is often: "Your workload does not reuse anything, so no cache will help."
The third break: the exchange rate is the worst in the system
Suppose popularity follows Zipf with α ≈ 1, over a corpus of N items. The top C items carry a fraction ln C / ln N of the traffic. So
- C (cache size)
- — the top k items (what you cache)
Each point of hit rate costs a constant multiplicative factor in cache size.
The popularity exponent derives this curve and the regimes on either side of it.
Compare the other tools you know:
| Tool | What it buys | Exchange rate |
|---|---|---|
| Compression | space | linear |
| An index | lookups | logarithmic (the lookup gets cheaper slowly as data grows) |
| A cache | hit rate | exponential in space (each point costs a multiple) |
This is why nobody chases the last few points of hit rate. Teams attack something else: the miss penalty (by adding a tier, a middle layer) or the miss count (by coalescing duplicate requests, or by prefetching).
There is a matching fact about latency: a cache with hit rate h leaves every percentile above h where it was, so at 95% the p99 is untouched (Miss penalty and the percentile you promised).
("Unregistered" means that nobody told the system a second copy exists, so no protocol keeps it in step.)