The recovery path is the normal path
The recovery path should be the normal path.
What it is
A recovery mechanism that has never executed under production conditions is an unverified assumption, and fault tolerance is made entirely of those. An untested path defaults to broken. The absence of observed failure is absence of evidence, not evidence of resilience; the only way to learn whether a rare path works is to make it not rare, deliberately.
From Measurement & Experimental Method › Chaos testing
The standard recovery algorithm for databases with mutable pages, ARIES, combines both directions: it redoes everything since the oldest dirty page, then undoes the transactions that never committed. How it works is part of Persistence & Durability. What matters here is why it is built the way it is: crashes during recovery are the normal case, not the edge case, so redo is idempotent (a record is applied only if pageLSN < recordLSN) and every undo step is itself logged and never undone. Recovery can crash and restart any number of times and still converge. It is the first strong evidence for the hypothesis that the recovery path should be the normal path.
From Fault Tolerance & Delivery Semantics › Rollback and undo
One item is in almost nobody's backup plan: an unverified backup is a hypothesis, not a backup. Restore testing is the only mechanism in this whole mechanism that actually samples the rare path. It is the activity with the highest information gained per dollar. A backup that silently fails to capture the data shows the same green dashboard as one that works, for years: backups are write-only until tested. The test is a periodic restore into a scratch environment, followed by a row-count and checksum comparison.
From Persistence & Durability › Backup and PITR
Where it appears
In 4 mechanisms. Each says what the mechanism is about, then how this principle shows up there, in its own words.
- Persistence & Durability
When a system says "yes, your data is safe", what exactly has it promised, and what did it have to do to keep that promise?
Crashes during recovery are the normal case, not the edge case, and ARIES is the standard demonstration of designing for that.
Read it inTwo liabilities: recovery and spaceBackup and point-in-time recoveryThe frame: what \"durable\" means
- Fault Tolerance & Delivery Semantics
What is the cheapest thing that can be safely done twice?
It is also a clean example of the principle "the recovery path should be the normal path": the code that re-commits after a crash is the same code that commits normally.
Read it inTransactional sinks and two-phase sink commitRollback and undo: where reversing stops
- Emergent Failure Dynamics
What happens when nothing is broken, every component is healthy, and the system's own response to stress makes the stress worse?
The general rule: recovery paths must not depend on the thing that failed.
- Observability & Operations
What is the smallest description of a running system from which we can find a fault, fast enough to matter, at a cost we can still pay while the system is failing?
The rule: capture the state, then mitigate.
Read it inThe incident lifecycle
Related
Made on the same pages most often.