What recovery forbids

What this article forbids

Reconstruction without a copy, a log or a recipe. There is no fourth source, and "highly available" is not one.
Low steady-state cost and short recovery distance together. Their product is roughly fixed; you move along the curve.
Faithful recovery of a nondeterministic computation. You get a valid state, not the state, and no mechanism detects the difference.
Safe promotion without enforcement at the shared resource. Agreement among reachable nodes says nothing about the one that is not answering.
Detection that is both fast and accurate. One knob, two errors, and no extra information to buy with.
Undo past an externally visible effect.
Termination of replay when drain rate ≤ arrival rate.
RTO independent of state size, once the state is larger than restore bandwidth × RTO.

Escapes and what each costs

Escape	Buys	Costs	Paid by
Retry	recovery with no stored artifact	wasted work; amplification when the fault is not independent	the dependency, at its worst moment
Keep the log	reconstruction of any state, no failure diagnosis needed	determinism you cannot verify; headroom that caps utilization	the operator, as an unexplained utilization target
Checkpoint	a bound on recovery distance	steady-state stall plus a collector for old checkpoints	the write path, continuously
Lineage	about 10⁻⁹ of the storage for full reconstructability	fan-in amplification; needs immutable sources	the scheduler, at the worst moment
Hot standby	RTO ≈ 0	a full duplicate, plus RPO = lag, plus fencing complexity	capital cost, always
Intent + scan	resolvable ambiguity at every boundary	2× write amplification; an index of in-flight work	the critical path, per operation
Immutability	undo as a pointer move; retires most of ARIES	space and cleaning work; retention bounds the window	the janitor, and the storage bill
Blacklisting	restored independence for retry	a classifier that can exclude the whole fleet	capacity, silently

Two escapes have no real cost and are reached for last: structuring the operation so the irreversible act is last, and making the recovery path idempotent by version comparison rather than by care.

Case, resolvedConstructed case · Wanjiru at Mediflow

Retries that flattened a nearly healthy database

In one week, retries from 400 instances flattened a nearly healthy database, a 3-hour backlog barely shrank after a restart, a replay gave different totals, a 10,000-task rollup never finished cleanly, and two machines briefly both believed they were the primary. Where it started: Recovery mechanisms →

Wanjiru and Dmitri fix the five symptoms one by one.

The retry storm. They add full jitter and a retry budget (retries limited to 10% of live traffic), and they make only the outermost layer retry, passing along a deadline so that deeper layers stop when the caller has given up.
The backlog that never shrinks. results-intake ran at 90% of maximum throughput. The drain rate was only 10% of maximum throughput (1/9 of the arrival rate), so a 3-hour outage would need about 27 hours to catch up. They set the utilization target from the recovery budget: to recover from a 3-hour outage within 1 hour (k = 3) they need maximum throughput of (3 + 1) = 4 times the arrival rate, so they plan for 25% utilization at peak, using more instances.
Different totals after replay. They find now() inside the aggregation and a lookup table that changed. They pass the event time in from the log and read the lookup table as of the event date. Replay now reproduces the original numbers.
The 10,000-task rollup. They accept that a failure is certain and rely on task re-execution with up to 3 attempts, plus node blacklisting with a decay time and a quota of at most 10% of nodes. They keep the shuffle files so a lost partition does not force the whole stage to rerun.
Two primaries. Dmitri adds a fencing token to the database failover: each promotion increments an epoch, and the storage layer rejects writes carrying a lower epoch. Because a wrongly triggered promotion would cost committed data, he keeps the failover of the main write primary manual and keeps automatic failover for the read replicas.