The tier-insertion rule
When is it worth inserting a new tier, and why are memory hierarchies spaced by orders of magnitude?
When is it worth inserting a new tier between two existing ones? Without the new tier, a miss costs t_below. With it, a request that reaches the new tier costs t_new + (1 − h_new)·t_below (it pays the new tier's lookup time, and then falls through with probability 1 − h_new). Improvement requires
t_new + (1 − h_new)·t_below < t_below ⟹ h_new > t_new / t_below
That one line explains the shape of every memory hierarchy:
| Gap | Ratio t_new/t_below | Break-even hit rate | Consequence |
|---|---|---|---|
| Tier 2× faster than the one below | 0.5 | > 50% | demanding; viable only where reuse is very strong |
| 10× faster | 0.1 | > 10% | comfortably achievable |
| 100× faster | 0.01 | > 1% | nearly free |
| Local NVMe (~100 µs) over object store (~50 ms) | 0.002 | > 0.2% | essentially unconditional |
So the order-of-magnitude spacing of memory hierarchies is not a hardware accident. It is the break-even condition. Tiers closer than about 4× cannot clear the bar for ordinary workloads, so they do not exist. Tiers 10× or more apart clear it easily, so the gaps get filled.
Two honest qualifications
- CPU tiers are spaced only about 4–5× apart, not 10×. There the constraint is physical (SRAM access time grows with array size), and the coherence overhead of another level is real, so the economics are tighter than the general rule.
- The dollar version of the rule has a different threshold. With per-request object-store pricing, a local disk cache is almost free in dollars per read. So an SSD tier over object storage is justified at hit rates far below what latency alone would require. This is an economic contingency, not a permanent constraint: if object-store first-byte latency and per-GET price both fell by a factor of ten, that tier would lose its reason to exist. It is the kind of claim, "today's pricing is mechanistically causal", that cost analysis must treat with care.
In one sentence: h_new > t_new / t_below. Order-of-magnitude tier spacing is the break-even condition, not a coincidence.