Summary and key takeaways
Two numbers. RPO (how much recent work a failure may take) is set by how fast the log reaches independent storage. RTO (time to resume service) is set by checkpoint distance and backup age, divided by replay speed.
The durability ladder. 0 client memory, 1 server memory, 2 OS page cache, 3 device cache, 4 media (honoured flush), 5 two failure domains, 6 a time-separated copy, 7 verified readable. "Durable" is a position, not a yes/no.
One-sentence summaries
- WAL: turn k random durable writes into one sequential durable write; pay with double writing and recovery work.
- Undo: the price of steal; also the version store, whose size is set by your slowest reader.
- Journaling: protects the filesystem's structure, not your bytes. Replace files with write temp, fsync file, rename, fsync directory.
- Group commit: one flush shared by many commits; natural batch = flush time × bandwidth ÷ record size; the batch window can be the previous flush.
- fsync:
O_DIRECTis not durability; a failedfsyncmust not be retried; every ack is only as true as the least honest layer below. - Checkpoints: sharp (stall), fuzzy (complexity, made possible by LSN-gated redo), incremental (permanent tax and a cliff). Recovery time ≈ interval × log rate ÷ replay rate.
- Snapshots: a garbage-collection policy on an indirection layer; nearly free to create, costly to keep, only logical protection until moved out of the failure domain; multi-volume snapshots must be atomic.
- Backup / PITR: base + log. Backup age sets restore time; log-shipping lag sets RPO; only a restore test proves it.
- Log compaction: O(keys) instead of O(events); turns a stream into a table; deletes need tombstones and global knowledge.
- Truncation: the log's floor is the minimum over five parties; a policy chooses who breaks: producer, consumer or budget.
- Torn-page protection: insurance against partial writes: double-write, full-page writes, or an atomic-write device.
- NVRAM / PLP: moves the durability point closer to the CPU and makes older amortization tricks into overhead.
- Checksums and scrubbing: convert silent loss to loud loss; must include block identity; scrub period + alert lag < retention of the last good copy.
- Replication, quorum, EC: W is the durable-copy count; independence is an assumption; EC saves space and costs repair amplification.
Key formulas
| Formula | Meaning |
|---|---|
| commit_rate = in-flight ÷ commit latency | the ceiling of a serialized log (Little's Law) |
| B ≈ T_flush × bandwidth ÷ record size | natural group-commit batch |
| RTO ≈ (checkpoint interval × log rate) ÷ replay rate | recovery time |
| stall = dirty pages × page size ÷ write bandwidth | sharp checkpoint cost |
| WAL amplification on first touch ≈ page size ÷ record size | full-page writes (8 KB ÷ 100 B ≈ 80×) |
| scrub period + alert lag < retention | when scrubbing can still save you |
| overhead = n ÷ k; repair reads = k chunks | erasure-coding trade |
Patterns that repeat
| Pattern | Seen in |
|---|---|
| Granularity mismatch creates amplification (and, here, correctness holes) | WAL, full-page writes, torn pages |
| A fixed cost per operation gives a natural batch size, and the fixed cost moves upward as hardware gets faster | group commit, checkpoint flush, NVRAM (media → syscall and latch → protocol) |
| Little's Law links latency and throughput | commit-rate ceiling |
| Costs are conserved, never removed, only moved | WAL, incremental checkpoint cliff, retention fork, scrubbing tax, torn-page insurance |
| Variance costs more than volume | sharp-checkpoint stall, ext3 compound-transaction fsync |
| Immutable storage grows an indirection layer plus a collector | undo / purge, CoW snapshots, tombstones |
| Independence is assumed everywhere and true almost nowhere | backup domains, drive batches, replication |
| Identical mechanisms at adjacent layers interfere | log on log on log |
| An ack is only as true as the least honest layer beneath it | fsync, drive caches; checksums are the only check |
| Every incremental scheme has a chain length trading capture against reconstruction cost | incremental backups, Spark lineage, snapshot chains |
| Retention is paid for by the live path, not the retainer | pinned undo, snapshots, slots, tombstones |
| Self-describing data cannot prove its own identity; it needs an outside anchor | ZFS parent checksums, T10 reference tags, manifests |
| Periodic stall events turn into continuous background rates | checkpoints, GC, compaction, backup |
| Base + log, in many forms | crash recovery, PITR, LSM, event sourcing |
One more observation: this mechanism's costs scale with bytes stored and written, not with anything exponential. A fixed rate meeting exponential growth does not appear here. So the pathologies are linear, and the cliffs are driven by thresholds, not runaway growth. Expect exponential effects when many participants must coordinate.
What this mechanism owns. It owns the durability decision: what must become non-volatile, where, and before which acknowledgment. If a mechanism changes what "yes" means or when it can be said, it belongs here. If it changes what happens after the crash (the replay algorithm, failover, repair), it is a recovery topic. If it is about how many machines must agree (protocols, elections, fencing), it is a coordination topic. If it is about how the bytes physically behave, it is a hardware topic. If it is scheduled housekeeping on a healthy system (expiry, vacuum, tiering, legal hold), it is a maintenance topic.