Five limits and their escapes
Which five things are impossible in memory management, and what does each way of living with them cost?
Dmitri's post-mortem
After the fourth incident, Dmitri Volkov writes a post-mortem for the whole team. He does not list bugs. He lists what the machine does not allow, and for each limit, what Voltara chose to do about it. Amara reads it and decides to turn it into a one-page note for the team. This article is that note, in full.
What this mechanism forbids
Five things are impossible. No mechanism here gets around any of them. Each one can only be worked around at a price.
- You cannot time-share capacity. The state is the work, so taking memory back destroys the work instead of delaying it. This is the root constraint, and everything else follows from it.
- You cannot know a footprint before producing it. The estimate inherits the optimizer's error distribution (Where the grant number comes from: the most important item in this article). That distribution has a heavy right tail, and no amount of statistics work removes it.
- You cannot account for memory you did not allocate. Any holder outside your allocator is invisible. In a system that runs user code (UDFs, libraries), such a holder always exists.
- You cannot make eviction free when eviction has correctness preconditions. Evicting a dirty page depends on the write-ahead log being flushed first. So the buffer pool's own latency is not bounded by the buffer pool.
- You cannot isolate bandwidth by bounding bytes. Capacity and bandwidth are different resources with different preemption costs. The mechanisms here handle only one of them: capacity.
The escapes, and what each costs
An "escape" is a way to live with a limit. None is free, except two.
| Escape | Buys | Costs |
|---|---|---|
| Take ownership from the layer below (buffer pool, arena, off-heap) | policy informed by knowledge the layer below lacks | you now own the bugs it used to prevent: dangling pointers, leaks, fragmentation |
| Declare intent instead of inferring it (scan ring buffers, registered consumers) | beats any adaptive policy | requires universal participation; the one non-participant sets your reserve |
| Ration at admission (grants, footprint admission) | attributable failure at zero cost | memory reserved and never used; queries queued that would have fit |
| Over-commit | utilization on ordinary days | correlated, total failure on the day the peaks coincide |
| Spill | completion instead of failure | the bandwidth ratio, the pipeline break, and a feedback loop into the pressure it relieves |
| Reformulate the operation (algebraic aggregates) | removes the need for the mechanism entirely | available only when the algebra permits; nothing to trade |
| Measure instead of estimate (grant feedback, ghost lists) | ground truth beats any estimator | only works for repeated plans and repeated access |
| Buy RAM | everything, briefly | the return curve of added memory is concave (each GB helps less than the last), and data grows faster than the budget |
The two rows with no real cost are declare intent and reformulate the operation. They are the ones practitioners reach for last. That is this mechanism's most useful practical lesson. Before you tune a policy, ask: can the caller tell me what it is going to do? Before you build a spill path, ask: can the operation be rewritten so that it does not need to hold the state at all?