Persistence & Durability
When a system says "yes, your data is safe", what exactly has it promised, and what did it have to do to keep that promise?
Who this mechanism is for: a developer who writes code, calls databases and knows basic SQL, but has never studied what happens between COMMIT and the moment data is truly safe on a disk. You do not need to know what a write-ahead log, a checkpoint, a quorum or a checksum is. Every technical term is explained the first time it appears, and again in the glossary at the end.
The story we will follow
Nadia is a backend engineer at Kestrel Pay, a small company that moves money between customer accounts. The core of the product is a ledger: a database that records every transfer, plus an event stream (an ordered list of "something happened" messages) that other services read. Her colleague Omar is the on-call engineer, the person who is woken up at night when something breaks.
Kestrel is about to get its first banking licence. The regulator asks one simple question: "If your server loses power at any moment, can you lose a transfer that you already confirmed to a customer?" During the audit period Nadia and Omar will meet eight problems, one at the start of each article below. Each problem looks like a bug. Each one is actually a consequence of how durability works. At the end of each article, we go back and solve the problem.
This is the main idea of this whole mechanism:
How this mechanism is organized
- 1
What durable means
Why is durability not a yes/no property, and which two numbers measure it?
Case40 confirmed transfers lost to a pulled cable
- 2
Logs and undo
How can a database say "yes" after one cheap write and still survive a crash?
Case40 ms per commit, and a zero-byte file
- 3
Group commit and fsync
How do we make that cheap write even cheaper, and where does the stack lie to us?
CaseStuck at 1,000 commits per second
- 4
Bounding the log
The log grows forever. How do checkpoints, compaction and truncation keep it finite, and who decides what can be thrown away?
CaseA full disk at 03:12, and a 25-minute recovery
- 5
Surviving your own mistakes
A replica copies a mistake in milliseconds. What keeps a way back in time, and what makes it independent?
CaseA DROP TABLE copied in milliseconds
- 6
When the storage lies
What if a write lands half-done, is acknowledged too early, or silently changes later?
CaseA balance that matches neither value
- 7
Durability across machines
How do copies on several machines raise durability, and what do they cost?
CaseA promoted replica, 300 ms short
- 8
Durability across a pipeline
When data passes through several systems, whose "yes" is the reader's promise, and for how long does it hold?
CaseA transfer the ledger kept and the stream lost
CaseWhat Kestrel can now promise
After that: a summary, hands-on labs, interview questions and a glossary.
One request: do not skip the numbers. This mechanism gives many small calculations (a 5 ms flush, a 20-second stall, an 80× amplification). The ratios are the real content.
What do I do, and what does each option cost?
Can I see it myself?
Kill vs. power loss: what survives?
Kill -9 a writer 50 times with and without fsync and count the acknowledged lines that survive. The power-loss half is not run here.
Break itPythonGroup commit: break the 1/latency ceiling
A log server syncing per commit against one with group commit, at 1 to 256 clients: commits per second and latency.
MeasurePythonGroup commit: measure fsync and the self-tuning batch
The cost of a durable write, 20 writes per flush, and pgbench with synchronous_commit on and off at 1 and 32 clients.
MeasurePython, DockerGroup commit and the square-root rule
Append 100-byte records with a flush after every 1, 10 and 100; then a timer batcher at λ = 2,000/s for waits W from 0.5 to 16 ms against W* = √(2·fsync/λ).
MeasurePythonWhat does fsync really cost on your machine?
Time 200-byte appends with no sync, with fsync, and on macOS with F_FULLFSYNC: the durability tax of your drive.
MeasurePythonCheckpoint interval versus WAL volume (Postgres)
Run pgbench for 120 s with checkpoints every 30 s and every 15 min, then without full-page writes; compare WAL bytes per transaction.
Real systemDocker, PythonRestore test and time-to-restore
Back up a 1 GB PostgreSQL database with WAL archiving, drop a table, restore to just before the drop, time it; then restore without the WAL.
Real systemDocker, PythonA checksum that proves identity
A toy block store with checksums next to the data and checksums in an index: which one catches a misdirected write and a lost write?
Break itPython
Can I reason about it under questioning?
Interview questions
Short questions on durability, such as write-ahead logs, fsync, checkpoints, backups, corruption and replication, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on durability, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Space vs. time
For example: compression, indexing, materialization, caching.
Durability vs. acknowledgment latency
How much must be confirmed before you say "yes".
Latency vs. throughput
For example: batching, group commit, micro-batch.
Precompute vs. compute-on-read
For example: materialized views, ELT vs. ETL, schema-on-read.
Ordering vs. parallelism
Global order costs serialization; parallelism costs order.
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Costs are conserved
Costs are conserved, never removed.
Independence is assumed
Independence is assumed everywhere and is true almost nowhere.
Variance costs more than volume
Variance costs more than volume.
Fixed cost per operation
Fixed cost per operation implies a natural batch size.
Retention is paid by the live path
Show all 14 principlesShow fewer
The recovery path is the normal path
The recovery path should be the normal path.
Granularity mismatch
Granularity mismatch creates amplification.
Immutable storage grows indirection
An immutable substrate grows an indirection layer plus a collector.
The end-to-end argument
Guarantees compose to the weakest link, and only endpoint guarantees are real.
The algebra decides
What can be skipped, pushed down, parallelized, merged or precomputed is determined by the algebra of the function, not the layout of the data.
Designate one write
Atomicity over n items is atomicity over one designated item.
Masking moves failure
Masking a failure moves it from a channel with a schema to a channel without one.
Guarantees have scope
Atomicity is bounded by the shared commit substrate.
Undeclared dependencies
An optimization that succeeds becomes an undeclared dependency, with leverage 1/(1 − h).