A distributed cache is not a latency tier

What does a distributed cache really buy, if a hit is slower than a local disk read?

"Local SSD, then distributed cache" is not a latency ordering

A common diagram lists the hierarchy as CPU cache → buffer pool → local SSD → distributed cache, which reads like steadily increasing latency. It is not. A distributed RAM cache is dominated by the network round trip at 0.3–1 ms. Local NVMe is about 100 µs. The distributed tier is typically 3–10× slower than the local SSD tier it is drawn beneath. By the tier-insertion rule, it cannot be justified on latency against the tier above.

It exists for three other reasons. Naming them correctly matters, because each leads to a different design.

Reason 1: Capacity pooling

If k nodes partition one keyspace (each key lives on exactly one node) instead of each node caching independently, the total distinct capacity is k·C. Under α = 1, hit rate rises by ln k / ln N. For k = 20 over a corpus of 10⁹ items: ln 20 / ln 10⁹ = 2.996 / 20.72 ≈ 0.145, so about +14 points of hit rate, independent of C.

But it costs a round trip on (k − 1)/k of the hits (a key is on the local node only 1/k of the time). So it pays only when the origin is much slower than the network.

Same design with a fast origin. t_origin = 2 ms.

Gain: 0.145 × (2 − 0.5) = 0.145 × 1.5 ≈ 0.22 ms.
Cost: still ≈ 0.43 ms.
Net: negative.

Hence keyspace-partitioned caching is everywhere in front of object storage and genuinely doubtful in front of a local database. The interesting question is not the hash ring (Making it fit). It is the ratio of the origin's latency to the network's latency. That ratio decides whether a distributed cache should exist.

Reason 2: Derived content

The remote tier usually holds objects whose miss cost is a recomputation, not a fetch. If making the entry costs 400 ms of computation, a 0.5 ms remote hit is a bargain, and the local-versus-remote inversion does not matter.

Reason 3: Surviving the process

A cache outside the application process survives deploys and restarts. This is not a latency argument at all. It is a defence against the cold-start pathology: after a restart the cache is empty, every request misses, and the backing store is overloaded (exactly what happened to Ifeoma). It is the reason a remote tier can be load-bearing in a system where the local tier is faster.

In one sentence: a distributed cache is slower than local disk. It buys pooled capacity (+ln k / ln N), recompute avoidance and restart survival, never latency.

Case, resolvedConstructed case · Ifeoma at Dawnpost

Doubling the cache did nothing

Doubling the article cache from 8 GB to 16 GB did not move the hit rate, a 95% hit rate left the 400 ms p99 untouched, and a colleague proposed a shared remote cache cluster for speed. Where it started: What makes data cacheable →

Ifeoma runs a key-hash-sampled reuse-distance histogram over a day of article requests. The MRC has a long plateau from about 6 GB to beyond 16 GB, and a cliff at about 24 GB, where the evening "most-read" working set fits. Her doubling from 8 GB to 16 GB moved along the plateau, so nothing happened. Part of the memory was also being filled by a nightly crawler (the one-shot component), which adds nothing to reuse. She decides on an admission filter first (Making it fit gives the mechanism), and treats the 24 GB cliff as the next sizing target instead of buying memory blindly.

She explains the p99 to Mateus. At a 95% hit rate, every percentile above the 95th is made entirely of misses. The p99 of 400 ms is the miss path: Postgres plus the archive object store. The cache cannot move it. She shortens t_miss: a local NVMe tier over the archive Parquet files (a hit at ~100 µs against 50 ms: break-even hit rate 0.2%) and a hedged second read after 80 ms.

She rejects the remote shared cluster as a latency tier. Her origin (Postgres, 2 ms) is only four times the 0.5 ms round trip, so by the pooling arithmetic the gain (0.22 ms) is smaller than the cost (0.43 ms). She keeps the idea only for derived fragments that cost 400 ms to recompute, and for restart survival.