Durability across a pipeline

When data passes through several systems, whose "yes" is the reader's promise, and for how long does it hold?

CaseConstructed case · Nadia at Kestrel Pay

A transfer the ledger kept and the stream lost

The ledger now survives a power cut, a lost rack and a wrong DROP TABLE. Then a customer calls: a transfer is in the ledger, confirmed, but no statement, notification or fraud check ever saw it. The payments service had committed the transfer and was restarted before it published the "transfer completed" event to the stream.

Two more problems follow in the same quarter. The statements service is switched off for 9 days during a migration; when it restarts, the first 2 days of its backlog are gone, although "the stream is durable". And the nightly archive job marks a day as done in the stream before its upload to object storage has finished; a crash in between leaves a day with no archive.

How it ends →

Every article so far asked what one system promises when it says "yes". A data platform is a chain: the ledger, an event stream, its consumers, an archive in object storage. The reader's question, "is this transfer safe?", is asked at the end of the chain, and every hop answers part of it.

This is the rule from fsync and flush barriers, "an acknowledgment is only as true as the least honest layer beneath it", applied to layers that are whole systems. It is where the durability question is most often answered wrong, because each team can truthfully say its own system is durable.

Each hop makes its own promise

Hop	What its "yes" means	Set by	Rung
Ledger commit	the WAL is flushed, and copied to a synchronous replica	`synchronous_commit` and friends	4–5
Producer → broker (Kafka)	`acks=0`: nothing was waited for. `acks=1`: the partition leader appended the record. `acks=all`: every in-sync replica has it	producer `acks`, topic `min.insync.replicas`	0, 2 or 5
Broker storage	the record is in the page cache of each replica; by default no broker flushes per message	`flush.messages`, `flush.ms` (off by default)	2 per machine
Consumer → sink	the consumer's bookmark (its offset) says the record was delivered	when the offset is committed	—
Sink: object store	a PUT returns success only after the object is stored redundantly; it is atomic, so there is no torn object	the storage class	5 or more

Two lines need a second look.

acks=all is rung 5 without rung 4. The record is in the memory of several brokers, none of which has flushed it. It survives the loss of one broker; it does not survive a power cut that takes all of them at once. That is a sound trade only when the replicas sit in independent failure domains, such as different racks or zones.

"All" means all in-sync replicas, and that set can shrink to one. When followers fall behind, they leave the in-sync set. With the default min.insync.replicas=1, acks=all then means "the leader has it": rung 2 on one machine, reported as success. min.insync.replicas=2 turns that case into an error the producer can see. And unclean.leader.election.enable (off by default) must stay off: turned on, it lets a replica that missed acknowledged records become the leader, and those records are gone.

Custody: who may forget, and when

Between two systems there are two "yes"es and a gap. The payments service commits the transfer, then publishes the event. A crash in the gap leaves the transfer durable in the ledger and absent from everything downstream, forever. No setting on either side closes it, because each side kept its own promise. The fix is to remove the second write: put the event in the ledger's own commit (transactional outbox), or read it from the ledger's log (log-based CDC). Either way the event becomes exactly as durable as the transfer. Why one fact must never be written to two systems is in Replay and dual-write avoidance.

Downstream, the same rule has a simpler form: a hop may forget its copy, or move its bookmark, only after the next hop's copy is durable. For a consumer the bookmark is its offset, and the order of the two writes is the delivery semantic: offset first loses a record on a crash, offset last repeats it, and storing the offset in the sink removes the choice (Consumer offsets and commit ordering). Durability's share is only the rule: the bookmark moves last.

Retention is a durability horizon

A log that deletes by time keeps its promise for its retention period and not a second longer. Kafka's default is 7 days (retention.ms). A consumer that is away longer finds its oldest records deleted; this is the "drop the laggard" policy from Log truncation and retention policies, and the loss is silent by design.

So a stream has two durability numbers, not one: what it survives (the rungs above) and for how long (its retention). The second is the one nobody writes on the architecture drawing. Size retention to the longest outage of any consumer you will tolerate, and measure consumer lag in time against that retention, not in records.

Case, resolvedConstructed case · Nadia at Kestrel Pay

A transfer the ledger kept and the stream lost

The transfer nobody downstream saw. The ledger and the stream were two writes with a gap. She moves the event into the ledger's transaction (an outbox table) and publishes from the ledger's log, so the event is exactly as durable as the transfer.

The 2 lost days. The stream deleted by time, and the consumer was away longer than the retention. She raises retention to 30 days and alerts when any consumer's lag passes half of it. She also sets acks=all, min.insync.replicas=2, three replicas in three racks, and checks that unclean leader election is off.

The day with no archive. The bookmark moved before the upload was durable. The archive job now commits its offset only after the object store's PUT has returned, and rewrites a day idempotently if it runs twice.

All the casesConstructed case · Nadia at Kestrel Pay

What Kestrel can now promise

Nadia writes the answer to the regulator as the ladder, one line per rung, each with the mechanism Kestrel bought and the test that shows it still works:

Rung 4, a power cut on one machine: a WAL forced on every commit with group commit, drives with power-loss protection, and a policy of stopping and recovering on any fsync error. Test: a pull-the-plug run in every acceptance cycle.
Rung 5, the loss of a machine or a rack: synchronous replication to another rack, replicas on different disk batches. Test: cut the primary's rack, promote, and count the confirmed transfers.
Rung 6, our own mistakes: a base backup every night, WAL archived every 60 seconds, kept in a separate account under object lock. Test: a restore every month.
Rung 7, silent corruption: page checksums that include the page number, a scrub every 7 days, an alert within a day, backups kept 30 days. Test: the scrub's own report, against the retention.
The chain: events written in the ledger's transaction, acks=all with min.insync.replicas=2, bookmarks moved only after the next copy is durable, 30-day retention with a lag alert.

Her answer: "A transfer confirmed to a customer survives a power cut, the loss of a machine or a rack, a mistaken command, and corruption found within the scrub window. It can be lost only if both racks in our zone fail together before the cross-region copy catches up; that window is the replication lag, and we report it." Omar signs it.