Summary and key takeaways

This mechanism is about what happens after something fails, and it has one frame: failure recovery is "act as if it did not happen" plus "make acting twice harmless". Silence on a network is slow or dead, a timeout is a guess, so every mechanism is built on that ambiguity.

This mechanism-wide accounting. Three quantities, three articles:

• RPO (recovery point objective): is there something to redo from? Durability and availability: where the thing to recover from comes from supplies it: replicas in separate failure domains, witnesses, repair, and backups with point-in-time recovery. Mediflow's 4 TB database needs about 5.6 hours to rebuild and about 6 hours to restore.
• RTO (recovery time objective): how long does the redo take? Recovery mechanisms governs it: retry with backoff, jitter and a retry budget; replay with a deterministic function and headroom (catch-up needs spare capacity); checkpoints at the optimal interval; lineage; fenced failover; re-execution and blacklisting; write-ahead intent; ARIES.
• Duplication factor: is the redo safe? Making a repeat harmless governs it: idempotency keys that name the intent, offsets stored with the effect, transactional sinks with a long enough timeout, epochs, sequence numbers, and a policy for poison pills.

Key takeaways

1. A timeout is a guess. Duplicates cannot be prevented at the sender, only absorbed where state is kept.
2. The real design question is: what is the cheapest thing that can be safely done twice? Prefer absorbing algebra, then deterministic addresses, then a dedup record.
3. Exactly-once delivery is impossible; exactly-once effect is achievable only inside what the system can commit atomically. Side effects such as SMS are outside that boundary, and the design should say so.
4. Commit the effect and the evidence of the effect as one write. Storing the offset in the same transaction as the row is the cheapest exactly-once.
5. Every dedup window is a bet on maximum duplicate delay, and the bet fails when the system is slow. Check sink transaction timeout > worst-case recovery time.
6. Costs are conserved and only moved: you wait, waste work, or weaken semantics. Poison-pill exits (stop, skip, quarantine) are a business decision.
7. Redundancy cannot be amortized. A replica's value comes from its failure domain, and rebuild time is the real durability variable. Independence is an assumption that real systems violate.
8. Replication protects against a lost copy; backup protects against wrong data. The metric is time-to-last-successful-restore, not backup success.
9. Degraded answers must be labelled, dependencies classified in advance, and recovery paths exercised, because an untested recovery path is an unverified claim. Chaos finds the undeclared dependency.

The two hypotheses of this mechanism. (1) The recovery path should be the normal path: idempotent commits, re-runnable epoch publication and routine restore tests make failure look like normal operation. (2) The end-to-end argument: a correctness property can be enforced only where the endpoints are; lower layers make it cheaper but never provide it. Examples are exactly-once effect versus delivery, readers honouring the epoch pointer, replication being faithful to a corrupting write, and degraded answers needing labels.

Wanjiru's outcome. Mediflow's pipeline now has something to redo from (Durability and availability: where the thing to recover from comes from), a way to redo it in a known time (Recovery mechanisms), and a way to make the redo safe (Making a repeat harmless).