Cache stampede (dynamics only)

Why does losing a cache turn into an instant overload of the backend?

A cache keeps a copy of expensive answers close to the caller. The hit rate h is the fraction of requests served from it. The backend sees only the misses: a fraction (1 − h) of the traffic. So removing the cache multiplies the backend load by exactly 1/(1 − h).

• With h = 90%, the amplification is 1/0.1 = 10×.
• With h = 99%, it is 1/0.01 = 100×.
• With h = 99.9%, it is 1,000×.

A 99% hit rate means the backend was sized, tested and staffed for 1% of front-end demand. Losing the cache is not a degradation; it is an instant 100× overload. The relationship is monotone and perverse: the better the cache, the larger the cliff behind it.

Two different loops live under this name:

1. The cold stampede is the capacity cliff above. A restart, an eviction wave, a flush or a new deploy empties the cache. It is bounded by the hit rate.
2. The per-key stampede is different. K concurrent readers miss the same key and send K identical backend requests, because a miss is not a lock. It is bounded by concurrency, not by hit rate.

This shows the principle that costs are conserved: they move, they do not vanish. A cache does not remove backend load. It moves load from a steady every-day state into a low-probability, high-magnitude event, and moves the risk from "known cost every day" to "unknown cost someday". The factor 1/(1 − h) is invisible in every measurement you take while the system is healthy.

Wayhop's surge-price cache has h = 99%. The pricing database is comfortable at 2,000 reads/s. At a peak of 20,000 requests/s, the database sees 1% of that, 200 reads/s. After a cache restart, all 20,000 reads/s go to a database that handles 2,000. That is 10× over capacity, and the cause was a restart, which Selin's dashboard marked as "healthy".