Why failure recovery is a different kind of problem

Why can a system not simply find out what happened?

The authority disappears

Think of any system that works normally. Somewhere there is a component that knows the truth: a log that either contains the commit record or does not, a transaction manager that knows whether it committed, an accountant that holds the byte count. When we reason about the system, we ask that authority.

Fault tolerance is the part where the authority goes away in the middle of a sentence, and the component that must decide what to do next cannot find out what already happened. This is not a weakness of some product. It is a property of asynchronous networks (networks where messages can take any amount of time): a machine that is slow and a machine that is dead produce the same observation, which is silence. Every failure detector is therefore a timeout, and a timeout is a guess.

Here is the constraint that governs this mechanism:

What stops mattering in this setting: whether the recovery path is fast. Recovery paths can be slow because they run rarely. But that permission is exactly what makes them dangerous: rarely run code is rarely tested (we return to this in Recovery mechanisms). What starts mattering: how many times an effect can be applied without changing the result.

The governing question is not "how do we avoid failure?". Failure cannot be avoided. The governing question is:

---

Three error classes, and only one is interesting

Every failure a component sees belongs to one of three classes. The mechanisms of this mechanism exist because the third class exists.

1. Deterministic or permanent. The same input will fail again. Examples: a message with a broken format (a "poison pill"), a schema violation, a bad partition key. Retrying is pure waste. The only choices are to skip, quarantine (set aside for a human) or fix.

2. Transient and independent across attempts. A dropped packet, a spot instance (a cheap cloud machine that the provider can take back) being taken away, a lock timeout. A retry will probably work. If each attempt fails with probability ε and attempts are independent, then n attempts all fail with probability εⁿ.

   Worked example. ε = 0.01 (1% of calls fail). Three attempts all fail with probability 0.01³ = 10⁻⁶, one in a million.

3. Ambiguous. The request may or may not have taken effect. The timeout fired, but a timeout tells you about your observation, not about the remote state.

The first two classes need almost no machinery. The third class generates nearly all of it. Retry, failover, task re-execution, replay and at-least-once delivery are all the same move: act as if the operation did not happen. Idempotency keys, deduplication windows, epochs, fencing and transactional sinks are all the same payment: make acting twice indistinguishable from acting once.

This is Wanjiru's payment call. The timeout means: "I did not hear back." The provider may have charged the card and the reply was lost (then a retry charges twice), or it may never have received the request (then not retrying means never billing). Both of her complaints come from the same ambiguity, and no retry loop, however well written, can remove it.

---

Three quantities: RPO, RTO and the duplication factor

The fault-tolerance mechanisms are organized into three groups, and each group bounds one quantity.

The first group is a mechanism of its own; this one covers the other two, and adds how a system keeps answering while a node is gone (Availability: masking a lost node). Read in a list, these look like three unrelated bags of techniques. Read as RPO, then RTO, then duplication, they are one argument: you chose how much you can lose; then you chose how fast you come back; and the third group is what the first two cost you in meaning.

---

The arithmetic that governs this mechanism

Two formulas decide most design questions.

Availability is dominated by repair time

When MTTR is small compared to MTBF, unavailability is approximately MTTR / MTBF.

Everyone who can chooses to divide MTTR.

MTTR itself has three parts: detect + decide + act. Detection usually dominates. A failover that takes 20 seconds to execute, but sits behind a 5-minute health-check window, is a 5-minute outage. The most valuable term in fault tolerance, detection, is really the job of monitoring and failure-detection systems rather than of the recovery mechanism. That is the first uncomfortable fact: the part you can build here is not the largest part of downtime.

Checkpointing has a clean optimum

A checkpoint saves state so that recovery does not start from zero. Taking checkpoints costs time; not taking them means you lose more work when a crash comes. Let:

• C = the cost (time) of taking one checkpoint,
• I = the interval between checkpoints,
• MTBF = average time between failures.

Expected overhead per unit time is:

The first term is the price of taking checkpoints. The second is the expected lost work: a failure arrives, on average, in the middle of an interval, so on average you redo I/2 of work, and failures happen once per MTBF. Minimizing this gives

I* = √(2 · C · MTBF)

The shape matters more than the numbers. The optimum moves as a square root. If you are wrong by a factor of 4 about the failure rate, the best interval changes only by a factor of 2, and the overhead near the minimum is flat. So checkpoint tuning is forgiving, which is why almost nobody tunes it and almost nobody suffers.