Summary and key takeaways
The seven articles, one sentence each:
- Feedback loops. A response to stress that adds load (retries, stampedes, timeouts, least-loaded routing) is a loop with gain, delay and sign. When the branching factor R is above 1, load grows from any starting point (4 tiers with 4 attempts each can multiply load 4^4 = 256 times).
- Metastable failure. In the band C/k < λ < C the system has two stable states, and the outage outlives its trigger. Fix k (the degraded-mode amplification), not the trigger. Recovery needs λ < C/k, so shed hard and readmit progressively.
- Cascading and correlated failure. Hidden dependencies, resource exhaustion, timeout ordering and shared fate break independence. Availability is capped by the least available shared component. Buy independence with cells, shuffle sharding, static stability and recovery paths that do not use what failed.
- Partial and gray failure. Fail-stop is an assumption that must be purchased. A slow node is worse than a dead one. Health is observed from inside the boundary it certifies, masking hides failures, and only external observers and data validation see delivered service.
- Death spirals. Systems defer work (GC, compaction, warm-up, catch-up) and the cost of paying rises with the debt. Autoscaling is a cost tool, restart is not a reset, and recovery is the peak load event.
- Overload defences. You cannot defend against overload by doing more work. Shed early and cheaply, propagate backpressure, budget retries, break circuits with probing and jitter, carry deadlines, hedge only the tail, and degrade before refusing. Every defence is a loop too.
- Synthesis. Six questions find these failures in design reviews.
The principles of this mechanism:
| Principle | One line |
|---|---|
| Concurrency = throughput × latency | Latency feeds in-flight work, which feeds latency (the GC loop, shedding triggers) |
| Costs are conserved | A cache, a buffer or a broker moves cost into a rare event, it does not remove it |
| Variance matters more than volume | Synchronized peaks, one slow member or one big item do the damage while the mean looks fine |
| A fixed rate meets exponential growth and loses | The decrease in AIMD must be multiplicative |
| Independence is assumed everywhere and true almost nowhere | It must be purchased with idle capacity, diversity, partitioning or staged rollout |
| Randomization can replace coordination | Jitter, two choices, shuffle sharding, salting |
| The regime-flip principle | An optimization flips sign at its regime boundary (retries, LIFO, generational GC, hedging) |
| The discard principle | Work is always discarded by someone; choose who, early and cheaply |
| The hidden-state principle | State-dependent cost means memory, two thresholds and no clean restart |
| The global-invariant principle | Deadlines, budgets and priority must travel in the request |
| Efficiency opposes independence | Sharing, standardizing and automating raise correlation |
| Masking moves failure to a channel without a schema | Every masking mechanism needs a counter |
| Self-reports measure the component, not the service | Only external observers see delivered service |
| Deferred work accrues interest | Every deferral needs a hard stop |
| Recovery is the peak load event | Compute recovery load and make part of the backlog droppable |
The six-question diagnostic: where is the loop (gain, delay)? What is k? What is shared? Who observes health, and from where? What work is deferred and does its cost rise? What does recovery cost, and is it provisioned?
The defence table in one line each: shedding and concurrency limits bound the store (ms), backpressure moves the store upstream, retry budgets and breakers cut the gain (s), jitter breaks correlation, deadlines remove wasted work, hedging buys the tail for about 5% load, brownout lowers cost per request, autoscaling is minutes and so not a defence.
The story in one paragraph. At Wayhop a 3-second pricing stall became a 40-minute outage (retry loop), outlived its cause (metastability), spread through an optional call and a shared config push (cascade), hid behind a green health check (gray failure), degraded itself through GC, compaction, flapping and replay (death spirals), and was finally contained by a short list of defences that each cost some work. Selin's checklist now stops the next one in the design review.