Retry amplification

Why do retries multiply load exactly when the system cannot afford it, and what bounds them?

The key fact is arithmetical, and it is about when the multiplier applies.

A retry policy of r retries produces a load multiplier between 1 and 1 + r. The multiplier depends on the failure probability f: when f ≈ 0 almost nothing is retried, so you pay 1×. When f → 1 every request is retried r times, so you pay (1 + r)×. The amplifier engages exactly when capacity is lowest. Load rises and capacity falls at the same moment. They are not two events; they are one event seen from two ends.

Composing over a call chain

Now take a chain of d tiers, each retrying r times. The multipliers multiply:

RelationshipRetry amplification across layers →

(1 + r)^{d}

1 + r (attempts per tier): $a$ — attempts each layer makes for one call
d (tiers): $L$ — layers that retry

Stated in 5 places →Principle: Local optimizations compose badly →

Ask the question that this mechanism asks of every design: what does this system assume that it cannot verify? Here: that it is the only retrier in the chain.

This is Selin's 18:05 incident. The database stalled for 3 seconds, calls timed out, every tier retried, and the database received a flood many times larger than normal at the moment it was least able to cope.

The structural fix

The fix is to stop expressing the policy per request. A per-request retry count cannot be made safe by tuning, because its multiplier is unbounded in f. A retry budget limits retries to a fraction of the successful request rate, so its multiplier is bounded by success volume: tied to evidence that the downstream service is alive.