What feedback loops forbid

Why can capacity not buy stability, which escapes exist, and what do they cost?

What this article forbids

You cannot buy stability with capacity. Capacity moves the threshold at which R crosses 1. It does not change the sign of the loop. If the structure produces R > 1 under saturation, every capacity increase buys you a larger, faster, more expensive cliff at a higher load. This is the single most useful idea here, and it is routinely ignored, because adding capacity can be bought and restructuring feedback cannot.

It also forbids detection by ordinary testing. Every amplification factor in this article equals 1 in the common case: retries do not fire, caches hit, queues stay short, timeouts do not trip. A load test at 40% utilization measures a different system from the one that fails, and the difference is not one of degree. The regime boundary is the object of study.

The escapes, and what each costs

Escape	Mechanism	Paid in
Bound	Queue caps, concurrency limits	Errors instead of latency; capacity planning becomes mandatory
Budget	Retry budgets, hedge budgets	Shared state (coordination); denial of legitimate retries
Randomize	Jitter, decorrelated jitter, two choices	Added latency; worse best-case balance
Propagate	Deadlines, cancellation	Viral plumbing; one non-participating hop defeats it
Discard	LIFO, deadline-at-dequeue, shedding	Unfairness; starvation of old work
Damp	AIMD, multiplicative decrease	Throughput left unused; slow ramp after recovery

There is no free escape, and none of them is a code change in one place. That is why these failures survive competent engineering.

Case, resolvedConstructed case · Selin at Wayhop

A 3-second stall, a 40-minute outage

At 18:05 a routine index rebuild stalled the pricing database for 3 seconds, and the site stayed down for 40 minutes; Bruno asked why not simply add 50% capacity. Where it started: Feedback loops →

Selin writes the post-mortem of the 18:05 incident in numbers. The chain gateway → booking → pricing → database has four tiers, each retrying three times, so (1 + 3)⁴ = 256 is the worst-case multiplier. The 3-second stall pushed delays past the 100 ms timeout; after that the database was busy doing work that clients had already abandoned (congestion collapse). Adding 50% more servers would only have moved the point where the loop crossed R = 1; it would not have changed the sign.

She proposes four changes. (1) A retry budget of 10% of successes at every tier, plus no retries on the lower three tiers when the upstream already retries. (2) Deadlines passed down the chain, so the database refuses work that cannot finish in time. (3) A bounded queue in front of the database (L_max = 2,000 reads/s × 0.2 s = 400), with discard at dequeue. (4) Jitter on the midnight jobs (spread over 60 s) and decorrelated jitter in the client library. For the surge cache she adds a rule that a restart must be done one node at a time, because a 99% hit rate means a 100× cliff behind it. And she replaces "send to the least-loaded dispatch node" with two random choices.