What death spirals forbid

Why is background work never free, and why is restart not a reset?

What this article forbids

It forbids treating background work as free. Compaction, collection, warming, rebalancing and catch-up all use the resource that serves the foreground, and their demand depends on foreground throughput. "It runs in the background" means "it competes, and it competes hardest when you are busiest."
It forbids best-effort deferral. If the cost of paying rises with the balance, the debt must be bounded by a hard stop. Every mature system has one: write stalls, OOM kill, admission control, watchdog. A deferral mechanism without a stop is a way to reach an unrecoverable state slowly.
It forbids the belief that restart is a reset. Restart is a move to the worst reachable state (cold, backlogged, herded). That is why restart is the operation most likely to turn a degradation into an outage.

Escapes and costs

Escape	Bounds	Paid in
Concurrency limit (caps L via Little's Law)	Live set, GC cost	Rejected requests; capacity planning required
Off-heap / arena allocation	Tracing cost of stable state	Manual lifetime management
Ingest rate limit below λ_max	Compaction debt	Throughput ceiling made explicit
Write stall	Unbounded L0	Total loss of write availability
Asymmetric control + hysteresis	Autoscaler oscillation	Over-provision on the up side
Reserved catch-up capacity	Recovery drain time	Idle resource
Drop or LIFO the backlog	Downstream replay storm	Data loss; broken ordering
Rate-controlled readmission	Cold-start collapse	Extended unavailability during warm-up
Staggered restart / warm peers	Fleet-wide cold state	Slower deploys

Every one of these is a deliberate reduction of peak throughput to get bounded behaviour. That is the price of this entire article.

Case, resolvedConstructed case · Selin at Wayhop

Flat traffic, and a p99 that climbs all afternoon

Search slowed by itself as GC spiralled, the pricing store fell behind on compaction, the autoscaler flapped, and a 10-minute outage replayed into a second one with a cold cache. Where it started: Death spirals →

The GC spiral. She sets a concurrency limit of 400 in-flight requests per search node (capping L by Little's Law), moves the large driver-location index off-heap, and keeps heap occupancy under about 60%. The p99 stops climbing.
Compaction debt. The pricing store's disk gives 1 GB/s of merge bandwidth and write amplification is about 50, so λ_max ≈ 20 MB/s. A nightly import was writing 60 MB/s. She rate-limits the import below 20 MB/s and alerts on the L0 file count, instead of waiting for the write stall.
Flapping. She sets scale-up to react after 1 minute and scale-down only after 15 minutes of low load, with separate thresholds, and adds a warm pool. The saw-tooth disappears. She also writes in the runbook that the autoscaler is a cost tool, not a defence.
Replay and cold cache. The 10-minute outage ran at ρ = 0.8, so the backlog took 10 × 0.8 / 0.2 = 40 minutes to drain at full speed. She now drains newest-first at a limited rate, drops the stale presence updates, and readmits traffic to a restarted pricing node in steps (5%, 10%, 25%, 50%, 100%) as its cache hit rate climbs.