Review a design for runaway failure
Some outages are not caused by a component dying, but by the system's own response to stress: retries that multiply load, a cache that empties, a health check that cannot see the problem. Adding capacity does not prevent them, because they are about ratios, not levels. In a design review, where could this system make its own failure worse?
You will end with the places where this design can amplify its own failures, and which number to measure for each.
Ask these in order.
Diagnose
Where is the loop?
Find any place where stress creates more work: retries on timeout, cache misses that load the database, autoscaling that adds load while it starts. Name its gain and its delay. If each unit of work in a degraded state creates more than one unit of further work, a disturbance grows without bound, however much headroom you have. You do not tune your way out of that; you change the structure that produces it.
How much does the degraded mode amplify load?
Measure k: how many times more work each request costs once the system is degraded (retries, cold caches, recomputation). A system with capacity C and arrival rate λ can stay stuck broken whenever C ÷ k < λ < C. With k = 3, every utilization above 33% is vulnerable. You can measure k while the system is healthy, and almost nobody does.
What is shared?
Shared dependencies (a database, a configuration service, a DNS resolver, one library version) correlate failures, and the system can be no more available than the least available thing everything shares. Name each one.
Who checks health, and from where?
A health check inside the boundary it certifies (a process reporting itself healthy) misses failures that only its users see: slow disks, partial network loss, wrong answers. Ask whether each check observes from the caller's side.
Where is independent work forced into one order?
One queue, one partition or one connection carrying unrelated work means one slow item blocks everything behind it (head-of-line blocking). Find each total order over work that did not need one.
What work is deferred, and does it get more expensive the longer it waits?
Compaction, cleanup and reconciliation postponed under load can grow costlier as the backlog grows, like interest compounding: an LSM-tree that falls behind on merging makes every read slower, which leaves less capacity to merge. Find each such debt and its limit.
What does recovery cost, and is anything set aside for it?
Recovery is not the reverse of failure: a restart arrives with cold caches and a backlog of retries, at the worst point. Check whether capacity, rate limits and a procedure exist for recovering, separately from normal running.