Scenarios

With 3 retries per tier over 4 tiers, the deepest tier can see up to 256 times its normal load. A 3-second stall leaves a queue of requests their clients have already abandoned, which the database needs minutes to drain.

A metastable failure: with an amplification of about 3, the band C / k < λ < C includes 50% utilization. The sustaining loop is retries plus queued work that no one is waiting for.

Cut offered load well below C / k (shed at the gateway, or turn off retries), wait for the queue to drain, then ramp up. Do not restart everything: a restart empties caches and adds a cold-start stampede.

A 10% retry budget on every client, jittered backoff, deadlines carried on each request so expired work is dropped when it is dequeued, bounded queues, and a circuit breaker.

Normally the database sees 1% of 20,000 = 200 requests per second. An empty cache sends all 20,000, ten times what it handles comfortably: the cache's leverage is 100.

• Warm the new cache from the old one before cutting over.
• Restart in rolling slices, so only part of the key space is cold at once.
• Coalesce requests, one database fetch per key, and allow serving stale values briefly.
• Add jitter to TTLs so keys do not expire together.
• Cap database concurrency with a bounded queue.

Coalescing makes the first request for each key wait; serving stale data shows old prices for a short time.

The sustaining effect is probably amplification, k, not capacity. More capacity moves only the top of the metastable band (C); k sets the bottom (C / k).

With k = 3 and capacity 2,000, the band is 667 < λ < 2,000; with 50% more capacity, 1,000 < λ < 3,000. A little better, but the band is still wide. Worse, more servers can mean more retries and more database connections.

Reduce k: retry budgets, propagated deadlines, load shedding. Measure k from a past incident: work done divided by useful demand.

With all 30 as hard dependencies, the ceiling is 0.999³⁰ ≈ 97.0%, so no amount of redundancy in your own code reaches four nines.

Turn dependencies into soft ones with fallbacks that are actually exercised (for example, recommendations become optional when rendering a page); remove dependencies from the critical path; isolate by cells; and measure the shared fate that makes the independence assumption false (a shared deploy pipeline, configuration, DNS). Use shuffle sharding where tenants are the source of failures.

Fallbacks need testing, cells cost multiplexing efficiency, and some dependencies, such as payments, cannot be made soft.

By Little's Law, concurrency = throughput × latency: 1,000 × 0.04 = 40 threads busy. If latency rises to 2 seconds, the same traffic needs 2,000 threads, so a 200-thread pool fills in about 0.1–0.2 seconds.

Bulkheads (a separate pool per dependency), timeouts shorter than the caller's, a bounded queue in front of each pool, and a circuit breaker.

A shared pool is efficient when everything is healthy, and becomes shared fate when one dependency is slow.

A gray failure: one slow node among 200. A search that fans out and waits for every node is slowed on every query by a single node 100 times slower, while only 0.5% of capacity is lost.

Build a table of latency by client and server. Compare each node with the median of its peers, and eject nodes above 3 times the median. Add an absolute alert for the case where every node is slow.

Cap evictions, for example at 20%, so a faulty checker cannot evict everything.

It helps only with random stragglers; against a node that is slow for a fixed reason it does nothing but add load.

The exit code is a self-report: it says the control flow finished, not that the data is right.

Make reconciliation the failure detector: an expected list of cities, row counts compared with the same weekday last week, freshness checks, and a counter for every value coerced to null.

Every masking mechanism, such as a permissive parser, a retry or a fallback, needs a counter, or it hides failure.

Compute cost, and expectations that drift, so review the thresholds regularly.

A failure detector cannot be correct in an asynchronous system: it sits on one curve trading false alarms against slow detection, and this one is set too far towards false alarms.

Tune the timeout with an adaptive (phi-accrual style) detector, require agreement from several observers rather than one, add fencing so the old primary cannot write after a failover, and use a watchdog so a hung node stops itself.

Capacity destroyed by false alarms, and more moving parts.

A garbage-collection loop. A pause raises latency; by Little's Law more requests are then in flight, so more data is live; heap occupancy ρ rises; and collection cost, about k ρ / (1 − ρ), rises with it, causing more pauses. Objects promoted prematurely defeat the generational collector. It is metastable.

Cap concurrency, move large state off the heap or into per-request arenas, and keep heap occupancy low.

The fleet-level effect, where nodes that pause get ejected and push their load onto the others.

Draining the backlog takes 10 × 0.8 / 0.2 = 40 minutes at full speed, and catching up at full speed overloads whatever is downstream.

Drain newest first at a limited rate, drop data that is too stale to matter, and reserve capacity for catching up. Readmit traffic in steps as the cache hit rate climbs, because a cold system's capacity may be 1/50 of a warm one's.

Staleness or lost data, in exchange for stability. Compute the recovery load explicitly before starting.

Admission control based on queueing delay, with cheap rejection and priority classes.

Deadlines carried along and checked when work is dequeued.

At one layer only, from a 10% token bucket, with jitter, and never in response to an overload signal.

Circuit breakers with probing, paired with adaptive concurrency limits; a tested ladder of brownout steps; hedging at p95 with a 5% budget, switched off under overload.

A game day to test how the defences compose. Autoscaling is a cost tool, not a defence, and every defence is itself a feedback loop.

Bulkhead the cluster: a guaranteed floor of capacity for interactive queries plus a shared pool for bursts. Give the backfill the lowest priority, and pace it to a fraction of capacity, adjusted by watching live query latency. Add cost-based admission for large scans.

The backfill takes longer.

At design time, mark each dependency as critical or optional, and give each optional one a reachable fallback.

Serve reads from stale data with a visible label ("up to 10 minutes old"); stop the ledger entirely (fail fast).

With chaos experiments, including injected latency (failing slow, not just failing), because a fallback that rarely runs is the least-tested code you have.