Fault Tolerance & Delivery Semantics
What is the cheapest thing that can be safely done twice?
Who this mechanism is for: a developer who writes services, calls APIs, uses a database and a message queue, and has seen a request time out in production, but has never studied how distributed systems recover from failure. You do not need any background in consensus, transactions or storage internals. Every technical term is explained the first time it appears, and again in the glossary at the end.
The story we will follow
Wanjiru is a backend engineer at Mediflow, a company of about 60 engineers. Mediflow runs a platform for a network of clinics. A clinic sends a blood sample to a laboratory. The laboratory sends the result back. The platform stores the result, charges the patient's card, and sends the patient a text message (SMS): "Your results are ready." The service that does this is called results-intake. It reads lab results from a message log (a durable, ordered queue), and for each result it writes a row to the patient database, calls an external payment provider, and calls an external SMS provider. Dmitri, the site-reliability engineer (SRE) who also looks after the databases, owns the replicas, the backups and the failover. Priya leads customer support and brings Wanjiru the complaints.
During one busy quarter, Wanjiru meets four problems, one at the start of each article below. Each problem looks like a bug. Each one is actually a consequence of a basic fact: when a message or a machine disappears, you cannot know what happened. At the end of each article we go back and solve her problem with the mechanisms of that part.
This is the main idea of this whole mechanism:
How this mechanism is organized
- 1
Why recovery is different
Why can a system not simply "find out what happened"? Which kinds of failure exist, and which quantities (RPO, RTO, duplication) do we trade?
CaseCharged twice, or never billed
- 2
Recovery mechanisms
How do we rebuild state after a failure: retry, replay, rollback, checkpoint, lineage, failover, task re-execution, intent logging, ARIES?
CaseRetries that flattened a nearly healthy database
- 3
Making a repeat harmless
Recovery creates duplicates. How do we make doing the same thing twice harmless (idempotency, exactly-once effects, commit ordering, poison pills)?
CaseTwo texts, two charges, one poison message
- 4
Availability: masking a lost node
How does a system keep answering while a node is gone: quorums, witnesses, repair, hinted handoff?
CaseTwo of three copies in one rack
After that: a summary, hands-on labs, interview questions and a glossary.
The thing you recover from (the durable log, the replica, the backup) is Persistence & Durability's subject, and it bounds how much work you can lose. Recovery mechanisms is the act of redoing (it bounds how long you are down). Making a repeat harmless is the guarantee that redoing is safe (it bounds how many times an effect may land). Availability: masking a lost node is how a system keeps answering while a node is gone, without redoing anything.
What do I do, and what does each option cost?
Choose a recovery mechanism
A step can fail part-way and must be recovered: compensate, keep a standby, recompute from lineage, restore a checkpoint, or both? Four questions decided by reversibility, the recovery-time target and whether the result can be recomputed.
ChooseMake a repeat harmless
Recovery will redo work, so an effect may land twice: use an operation that absorbs repeats, store progress with the effect, commit through the sink, or deduplicate at the receiver? Four questions, cheapest first.
Choose
Can I see it myself?
Checkpoint interval optimum
A 30-day job with Poisson failures: sweep the checkpoint interval, find the optimum against √(2·C·MTBF), then get the MTBF wrong by 4×.
SimulatePythonRetry storm: no jitter versus full jitter
400 clients retry against a server that was down for 2 s: exponential backoff with no jitter, equal jitter and full jitter.
SimulatePythonThe catch-up condition
A 60-second outage at 500 arrivals per second, recovered at throughputs from 525 to 2,000: recovery time against utilisation.
SimulatePythonReplay determinism check
An aggregation run twice over the same events, with now() and a mutable lookup, then with event time and a versioned lookup: compare output hashes.
MeasurePythonDuplicates versus an idempotency key
A small payment service, 1,000 charges and 20% lost replies: a new key per attempt, an intent key with a stored outcome, and suppression with an error.
Real systemPythonDedup window size and the Bloom filter trap
The memory of exact deduplication for an hour and a week at 100,000 keys/s, then a Bloom filter alone and in front of an exact set.
MeasurePythonIdempotent sink with a conditional insert
Redelivered batches into SQLite: a plain insert duplicates, a keyed conditional insert does not, and check-then-insert races.
Break itPythonCommit ordering and the offset in the sink
A consumer adds 1..10,000 to a total and crashes 20 times: position saved before processing, after, and in the same transaction.
Break itPythonAuto-commit versus commit-after-process
Kafka auto-commit against commit-after-process, each killed once mid-run: silent loss against a bounded number of duplicates.
Real system · Break itDocker, Python with kafka-python
Can I reason about it under questioning?
Interview questions
Short questions on fault tolerance, such as error classes, retries and checkpoints, failover, idempotency, exactly-once effects and redundancy, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on fault tolerance, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Precompute vs. compute-on-read
For example: materialized views, ELT vs. ETL, schema-on-read.
Durability vs. acknowledgment latency
How much must be confirmed before you say "yes".
Cost as a substitutable resource
Money buys position on every other axis; the only axis denominated in a different unit.
Space vs. time
For example: compression, indexing, materialization, caching.
Consistency vs. availability vs. latency
For example: quorums, replication modes.
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Independence is assumed
Independence is assumed everywhere and is true almost nowhere.
Designate one write
Atomicity over n items is atomicity over one designated item.
Fixed cost per operation
Fixed cost per operation implies a natural batch size.
Costs are conserved
Costs are conserved, never removed.
Granularity mismatch
Granularity mismatch creates amplification.
Show all 16 principlesShow fewer
Variance costs more than volume
Variance costs more than volume.
The algebra decides
What can be skipped, pushed down, parallelized, merged or precomputed is determined by the algebra of the function, not the layout of the data.
The end-to-end argument
Guarantees compose to the weakest link, and only endpoint guarantees are real.
The recovery path is the normal path
The recovery path should be the normal path.
Immutable storage grows indirection
An immutable substrate grows an indirection layer plus a collector.
Which way the error points
Sound approximations
When exactness is too expensive, use a sound approximation.
Little's Law
Concurrency = Throughput × Latency.
Queues explode near full load
Response time explodes as utilization nears 100%.
Randomization replaces coordination
Randomization can replace coordination.
Local optimizations compose badly
Identical mechanisms at adjacent layers do not add up. They interfere.