Summary and key takeaways
The six parts, one sentence each:
- Memory cannot be time-sliced, and the kernel's swap chooses victims by recency, so serious engines take the decision back. RAM has four claimants: buffer pool, working memory times concurrency, OS page cache, runtime overhead.
- Ownership: arenas, off-heap, GC tuning, pooled buffers, dictionaries and NUMA placement each trade a cost for a cost somewhere else.
- Residency: LRU, Clock, LRU-K, 2Q and ARC approximate an unknowable future; declared intent (ring buffers) beats inference; pins and dirty pages limit eviction.
- Rationing: grants and admission move failure to the start; the grant is an estimate with a heavy tail; pressure signalling is cooperative; byte limits do not isolate bandwidth.
- Degradation: spill replaces the algorithm, costs the bandwidth ratio rather than the fraction spilled, fails on one oversized key unless you use sort or algebra, and ends in a kill decision.
- Synthesis: five limits, eight escapes with prices, and two rules.
The five limits (nothing evades them): you cannot time-share capacity; know a footprint before it exists; account for memory you did not allocate; make eviction free when it has correctness preconditions; or isolate bandwidth by bounding bytes.
Key formulas and numbers:
| Formula | Meaning | Seen in |
|---|---|---|
| GC cost ≈ live set ÷ headroom | collector work per byte freed | JVM tuning, off-heap |
| passes = ⌈log_{M/B−1}(N/M)⌉ | merge passes are a staircase in memory | grants, Memory grants and reservations per operator |
| one-pass capacity = M²/B | memory enters quadratically | hash and sort spill |
| time ∝ (1 − f) + 2fr | spill slowdown; r = 50 gives about 6x at 5% | What spilling actually costs |
| R = S / (1 − ρ) | response time near full utilization | spill device, memory controller |
| L = λW (Little's Law) | spill raises residence time, so raises in-flight queries | What spilling actually costs |
Patterns that repeat:
- Granularity mismatch, at every scale from an allocation size class to a whole node, including failure granularity (process versus query).
- Costs are relocated, never removed: grants move failure to admission, victim selection moves loss to a chosen query.
- The tail, not the mean, sizes grants, picks the join algorithm and decides the reserve.
- Independence assumed, but false: predicates correlate upstream, peaks correlate downstream.
- Declare intent, do not infer it, and measure instead of estimate: the two escapes that cost nothing (with reformulating the algebra).
- Where preemption is expensive, ration at admission, which forces an estimate; where it is cheap, schedule continuously.
Practical checklist:
- Write the four claimants of RAM and make sure two substitutes are not double-buffering.
- List what cannot spill; that is your kill inventory.
- Compare each grant with the actual peak; add feedback.
- Grant in steps, count bytes at admission, hold a reserve for invisible holders, use two watermarks with hysteresis.
- Compute
rfor the real spill device; compress spill; gate concurrent spillers. - Check the aggregate's algebra before building anything else.
- Fail a query before the OS fails a process, and connect the kill path to grant feedback.
- Remember that bandwidth is the one resource nobody isolates.
Amara's closing note (Voltara): ration before the work starts, trust measured peaks over the optimizer, spill for survival and not speed, reduce before writing, and kill a query before the kernel kills a process.