Emergent Failure Dynamics
What happens when nothing is broken, every component is healthy, and the system's own response to stress makes the stress worse?
Who this mechanism is for: a developer who writes services, calls other services over a network and has seen an outage or two, but has never studied queueing theory, control theory or failure analysis. You do not need any background in distributed systems. Every technical term is explained the first time it appears, and again in the glossary at the end. This mechanism is about failures that have no bug and no broken part: the system fails because of how its own parts react to each other.
The story we will follow
Selin is a backend engineer at Wayhop, a ride-hailing company. Wayhop serves about 6 million rides per day in 40 cities. A rider opens the app, the request passes through an API gateway to a booking service, and the booking service calls a pricing service, a dispatch service (which matches riders to drivers), a maps service, a payments service and, in the render path, an optional "ride again?" recommendations service. Prices come from a pricing database, with a cache in front. Two hundred "search nodes" answer the question "which drivers are near me?". At night a pipeline turns the day's trip events into reports for finance and operations. Bruno is the engineering director. He asks the hard questions: "Why did a 3-second blip cost us 40 minutes?", "Why not just add servers?", "Why not just restart it?" Wei is the data engineer who owns the nightly pipeline.
Over four months Selin will meet seven incidents, one at the start of each article below. Each one looks like a bug. Each one is actually a dynamic: a behavior that appears only when the parts of a healthy system react to each other. At the end of each article we go back and explain what happened and what to change. The numbers in the story are illustrations. The mathematics in the articles is real.
This is the main idea of this whole mechanism:
How this mechanism is organized
- 1
Feedback loops
Why does a small slowdown turn into a flood of extra work?
CaseA 3-second stall, a 40-minute outage
- 2
Metastable failure
Why does an outage continue after its cause is removed?
CaseRolled back, and the site stayed down
- 3
Cascading failure
Why do unrelated things fail together, and why do replicas not help?
CaseAn optional call took booking down
- 4
Gray failure
Why can the system not tell that a component is bad, and why is "slow" worse than "dead"?
CaseOne slow node, and every search slow
- 5
Death spirals
How can a system degrade itself when no client misbehaves?
CaseFlat traffic, and a p99 that climbs all afternoon
- 6
Overload defences
How do we force the loop to shrink instead of grow, and what does each defence cost?
CaseSeven postmortems and twenty fixes
- 7
Six questions
What does this whole mechanism forbid, and what six questions should we ask of any design?
CaseSeven incidents, one page
After that: a summary, hands-on labs, interview questions and a glossary.
There is an order behind the articles. The first articles are increasing distance from the client. Feedback loops are driven by clients (they retry). Death spirals are driven by the system itself (it garbage-collects and compacts). Overload defences come last because they are the answer to the rest.
What do I do, and what does each option cost?
Review a design for runaway failure
Seven questions for a design review that find the failures a system causes itself: feedback loops, degraded-mode amplification, shared dependencies, blind health checks, head-of-line blocking, deferred work and unpriced recovery.
DiagnoseRecover from a stuck overload
The trigger is gone but the system stays down: why shedding a little load and restarting both fail, how much load must really go, and the five-step procedure that gets out.
Design
Can I see it myself?
Measure retry amplification and a retry budget
A backend stalls for 3 seconds; clients time out and retry. Peak offered load with and without a 10% retry budget.
SimulatePythonFind the metastable band
A queue whose work per request triples when it is long: healthy and degraded states at the same load, and the shed-and-ramp exit.
SimulatePythonAvailability ceiling and the benefit of cells
30 hard dependencies at 0.999 against a^N, soft dependencies with a fallback, and a poison tenant with no cells, 10 cells and shuffle sharding.
SimulatePythonBulkheads: one slow dependency, one shared pool
One pool of 10 threads shared by a dependency that hangs and one that is fast, against two pools of 5.
SimulatePythonGray failure: dead node versus slow node
A 200-node scatter-gather query with one dead node, one slow node, peer-relative ejection, an all-slow fleet and an eviction cap.
SimulatePythonAutoscaler flapping and the effect of dead time
An autoscaler targeting 60% utilisation with a provisioning delay of 0, 2 and 5 minutes, then with hysteresis and slow scale-down.
SimulatePythonRecovery time arithmetic and replay storms
Stop a queue for 60 s at ρ = 0.5, 0.8 and 0.9, time the drain, watch the catch-up overload a downstream stage, then drop stale items.
SimulatePythonA circuit breaker that protects the caller
A dependency that hangs for 30 s: without a breaker every thread blocks; with one the service answers fast and recovers.
SimulatePython
Can I reason about it under questioning?
Interview questions
Short questions on how failures grow, such as feedback loops and retry amplification, metastable failure, cascades, gray failure, contention, death spirals and overload defences, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on how failures grow and how to stop them, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Fail-fast vs. fail-degraded
Reject or continue when something is wrong.
Isolation vs. utilization
For example: multi-tenancy, resource pools, spot capacity.
Consistency vs. availability vs. latency
For example: quorums, replication modes.
Coupling vs. coordination cost
For example: 2PC vs. sagas, schema contracts vs. schema-on-read.
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Local optimizations compose badly
Identical mechanisms at adjacent layers do not add up. They interfere.
Absence is not evidence
Absence of an event is not absence of a failure.
Hidden state breaks replay
A replayable pipeline is a pure function of its retained inputs.
Costs are conserved
Costs are conserved, never removed.
Independence is assumed
Independence is assumed everywhere and is true almost nowhere.
Show all 12 principlesShow fewer
Queues explode near full load
Response time explodes as utilization nears 100%.
Randomization replaces coordination
Randomization can replace coordination.
Variance costs more than volume
Variance costs more than volume.
Masking moves failure
Masking a failure moves it from a channel with a schema to a channel without one.
The end-to-end argument
Guarantees compose to the weakest link, and only endpoint guarantees are real.
Little's Law
Concurrency = Throughput × Latency.
The recovery path is the normal path
The recovery path should be the normal path.