Timeout cascade
Why must timeouts decrease down a call chain, and why can local configuration not keep them coherent?
Timeouts must form a strictly decreasing sequence down a call chain.
- If the caller's timeout is shorter than the callee's, the caller gives up while the callee is still working. This is the wasted-work problem, now structural instead of accidental.
- If the caller's timeout is longer than the callee's worst case (including the callee's own retries), the caller holds its resources for the difference, which feeds the exhaustion cascade above.
The correct rule at depth i is that the timeout must exceed the full budget of everything below it, with retries multiplied in.
(If each tier allows one attempt plus 3 retries, the bound is 4³ = 64 s. Either way the consistent budget is far larger than anyone configures.)
Here is the general shape, and it recurs: timeout ordering is a global invariant, and each team can edit only a local configuration. No amount of local correctness produces it, because correctness is a property of the composition and nobody owns the composition. This is the same structure as the retry-budget problem: each tier's retry count is safe alone and lethal composed. The only fixes are:
- carry the invariant in the request itself (deadline propagation), or
- enforce it in a component every hop shares (a service mesh, a common RPC library with a mandatory budget).
Both are coupling costs, paid in plumbing and in a mandated shared runtime.
This is the global-invariant principle: a global invariant cannot be maintained by local configuration. It must be carried in the request or enforced by a component every participant shares. Timeout ordering, retry budgets, deadline discipline, trace context, priority and tenant identity all have this shape. Each participant can be locally correct and the composition still violated, because no participant can observe the composition. Predictive form: wherever a property must hold across a chain but is configured per hop, expect it to be violated in production, and expect the organizational resistance to be the real obstacle, since both fixes impose coupling on teams that were deliberately decoupled.