GC death spiral

How does garbage-collection cost feed latency, and latency feed the live set?

A garbage collector (GC) finds memory that the program no longer uses and makes it available again. A tracing collector works by walking all live (still reachable) objects. Dead objects cost nothing to find, because the collector never visits them. So the cost of a collection is proportional to the live set, not to the garbage.

Let H be the heap size and L the live set. One collection cycle costs about k·L (k is a constant for the collector) and frees (H − L) bytes. So the cost per byte reclaimed is

This is the same 1/(1−ρ) curve as queue delay (the queueing wall). It is not an analogy: heap occupancy is utilization, and GC overhead diverges (grows without limit) at full occupancy for the same reason queue delay diverges at full utilization.

Closing the loop

Little's Law (concurrency = throughput × latency) supplies the missing edge. Most of the live set is request-scoped state (data that exists while a request is in flight). So the live set is proportional to L = λ·W, the number of requests in flight. A rise in latency mechanically raises the live set. The cycle:

GC pause → request latency rises → more requests in flight → live set grows → ρ rises → each collection costs more and frees less → longer pause.

Every arrow is forced by arithmetic. Nothing is a bug. This is the cleanest self-sustaining loop here, and it is metastable as in Metastable failure: the trigger (a burst, a slow dependency) can be removed and the system stays at high occupancy, because the occupancy is now held up by its own latency.

Why generational GC fails here

Generational collection is the standard defence. It relies on the weak generational hypothesis: most objects die young. So short-lived request garbage is collected cheaply in a small "nursery" area. But an object survives a nursery collection if its request is still in flight, and under load requests stay in flight longer. So premature promotion (moving objects to the old generation too early) depends on latency. The cheap collector turns into the expensive one exactly when you need it. This is the regime-flip principle: an optimization is conditional on a regime, and at the boundary its sign flips.

The fleet-level version

It is worse across a fleet. A GC pause makes a node miss heartbeats. The failure detector (Gray failure) ejects it, or the load balancer routes around it. Its traffic moves to its peers. The peers' in-flight counts rise, so their live sets rise. A GC spiral spreads from node to node.

Escapes

All escapes are the same move: bound the live set, not the heap.

• Concurrency limiting is the cheapest. Capping in-flight requests caps L directly by Little's Law.
• Move large, long-lived state off-heap (outside the garbage-collected area; this is why data engines moved to off-heap and Arrow-style buffers).
• Per-request arenas: allocate everything for one request in one block and free the block at once. Lifetime is explicit and reclaiming is a pointer move.

All three refuse to let the runtime infer lifetime, which it can only do by tracing, and tracing is the expensive thing.