Keep a cache honest
A cache sits in front of a store whose data changes. Every entry is a claim about the past, true when it was filled. How do writes reach the cache, how stale may an entry get, and how do you keep expiries and restarts from flooding the store behind it?
You will end with an invalidation design, a bound on staleness, stampede protection for hot keys, and a warm-up plan.
Work through these in order: the first two can make the rest unnecessary.
Work through it
Find out whether anything writes around the cache
If every write goes through the cache: it is a coherent buffer, not a cache, and you need none of what follows. Many teams skip this question and add a TTL they never needed.
Make the data immutable if you can
If the data can be made immutable, with its version or content hash in the key: do it. An entry can then never be stale, and invalidation disappears.
On a write, delete the entry; never overwrite it
Measure how often a key is read soon after it is written, and use that to decide whether writes populate the cache or bypass it. Either way, a write deletes the entry; it never writes the new value into it.
Why
Nothing orders cache writes against store writes. If two writers update the same key and the slower one writes the cache last, the cache keeps the older value forever. A delete has no such race: the next read fetches whatever the store now holds.
Keep a TTL, even with explicit invalidation
An expiry time is the floor under lost invalidation messages, writers you do not know about, and the race where a read refills an entry just after it was invalidated. It is not an alternative to invalidation. Know its cost: a TTL forces a miss on every hot key every period, whether or not anything changed: a million hot keys at 60 s is about 16,700 forced misses a second, even with no writes.
Why
One lost invalidation message means unbounded staleness. The TTL bounds it. But its cost grows with the size of the cache, not with how often the data changes, which is backwards; that is what step 5 fixes.
If entries are large, verify instead of refetching
If payloads are large compared with a round trip: add verification: ask the store "has this changed?" (an ETag or conditional fetch) and refetch only if it has. Freshness then costs a round trip, not a transfer. It is the instrument most often missing.
Decide whether entries must agree with each other
If several entries must be consistent with each other (a page assembled from many cached parts): designate one version cell, such as a snapshot pointer, and key every entry to it. If you cannot: say plainly that cross-entry consistency is unavailable, and stop treating its absence as a bug.
Why
Every system that gives consistency across many cached entries has one designated version cell; per-key TTLs and per-resource ETags never do.
Protect hot keys from stampedes
When a hot key expires, every request for it recomputes it at once: duplicate recomputes = requests per second for the key × seconds to recompute. At 1,500 requests a second and a 2-second query, that is 3,000 identical queries. If that number is above 1: add protection, paying in one of three currencies:
- Coalescing: one request recomputes, the rest wait for it (pays in waiting).
- Probabilistic early expiry: each request may refresh slightly before expiry (pays in a little extra work, needs no coordination).
- Serve stale while revalidating: return the old value while one refresh runs (pays in staleness).
Why
The duplicate queries slow the store, which makes recomputes longer, which adds more duplicates. Past the store's capacity the key never refills, and the system stays down after the trigger has gone.
Jitter every TTL
Entries filled together (by a deploy, a warm-up, a batch load) with the same TTL expire together, stampeding the whole key space. Use TTL × (1 ± 0.1 to 0.2), always.
Plan the warm-up from the store's capacity
After a restart the cache is empty, and the store is sized for the load the cache used to absorb. Admit traffic at store capacity ÷ (1 − current hit rate), raising it as the hit rate climbs. With a store that handles 1,500 reads a second: 1,500/s while empty, 3,000/s at a 50% hit rate, the full 30,000/s only at 95%. Replace any fixed-duration warm-up with this.