Scenarios

A write-ahead log, flushed once per group of commits, not once per transaction. With a 1 ms flush and no batching, the ceiling is 1,000 commits/s (Little's Law), so use group commit: at 20,000/s each flush must carry at least 20 commits, which is easy under load.

Drives with power-loss protection (an fsync of about 20 µs). Verify the claim with a test that pulls the power.

For zero data loss when a whole machine dies, replicate synchronously to another failure domain: about 0.5 ms of round trip plus the replica's flush (synchronous commit latency).

Commit latency of about 1–2 ms, against about 0.1 ms with asynchronous replication.

A stalled replica blocks commits, so decide in advance who gives way and after what timeout. An fsync error must stop the database and trigger recovery, never a retry.

Group commit, power-loss-protected drives, and a synchronous replica in another rack, plus an asynchronous replica in another region, documented honestly as losing up to its replication lag.

Recovery time ≈ checkpoint interval × log rate ÷ replay rate. Measure each term: for example a log rate of 28 MB/s, a replay rate on a cold buffer pool of 14 MB/s, and a 12-minute checkpoint interval, giving 24 minutes.

1. Shorten the interval to about 2.5 minutes, 5× less log to replay, but expect more full-page images and therefore more log volume: measure it.
2. Use adaptive or incremental flushing so the checkpoint never falls far behind, watching for the throttling cliff when flushing cannot keep up.
3. Replay faster: more memory, prefetching during the analysis pass, faster random-read storage.
4. If downtime must be near zero, run a hot standby that replays continuously; the recovery time becomes the failover time.

Shorter intervals trade recovery time for write amplification.

The log can be truncated only to the minimum position among everything that still needs it.

1. A replication slot or CDC consumer stopped advancing.
2. A long-running transaction is holding old row versions (bloat).
3. The archive command is failing, so the log cannot be recycled.
4. A forgotten copy-on-write snapshot is forcing new allocations.

Slot lag in bytes, the age of the oldest transaction, archive status, and the list of snapshots.

Alert on the reader's metric (slot lag, age of the oldest transaction), cap the log a slot may retain (an explicit choice of whether the producer, the consumer or the budget gives way), and time out sessions left idle in a transaction. Retention is paid for by the live system.

Replication copied the drop within milliseconds. You need a copy separated in time, not just in space.

If a base backup and archived log exist: restore the base backup to a new server, replay the log to just before the DROP, and export the table. The restore takes the base restore plus replay of all log since the base, so daily base backups keep it short; the data loss is bounded by the archive lag (for example 60 s).

Keep backups independent of the production account (a separate account, object lock), and test restores on a schedule: an untested backup is a hypothesis. Add a delayed replica, for example one hour behind, as a cheap safety net.

Check identity as well as content: per-block checksums stored in the parent pointer, or the page number inside a checksummed page. Verify them on every read and with a background scrub.

On a mismatch, fetch the block from another replica and rewrite it.

Scrub period + alert lag < retention of the last good copy: with 14 days of retention, scrub every 7 days with a 1-day alert lag. Budget the bandwidth: 10 PB every 7 days is 10 × 10¹⁵ ÷ 604,800 s ≈ 16.5 GB/s of reads.

Errors are correlated within a batch of drives, so place replicas on different batches and racks.

A bug in the verifier can cause panics, so handle a mismatch by repairing, never by crashing everything.

The access pattern and the object sizes.

Storage: 3× replication needs 6 PB; 10+4 erasure coding needs 1.4× = 2.8 PB. Repair: replication copies one byte per byte lost; erasure coding reads 10 bytes per byte lost, so one failed 16 TB disk causes about 160 TB of repair reads.

Replication for hot, small or latency-sensitive objects. Erasure coding for cold, large objects such as archives and old logs, possibly with local reconstruction codes to cut repair reads. A hybrid works well: replicate recent data, and convert to erasure coding once it cools.

To get the stated tolerance, spread the 14 chunks over at least 14 independent failure domains.

The point where a write counts as durable has moved. A RAID controller's battery or capacitor degraded and it fell back to write-through, or a drive without power-loss protection replaced one with it: about 20 µs against 1–2 ms per flush, a 50–100× difference.

Measure fsync latency directly, check the controller's cache policy and battery status, and check whether the drive model has power-loss protection.

Replace the battery, or accept lower throughput while staying safe. Do not "fix" it by disabling flushes: that silently gives up durability.

Monitor fsync latency and the cache-policy state, which would have caught this immediately.

The audit consumer needs the history of changes, so key-based compaction would silently break it. Use time- or size-based retention for the audit topic, and a separate compacted topic for consumers that want current state.

Decide who gives way when the audit consumer falls behind: the producer (it stalls), the consumer (it loses data) or the budget (tiered storage).

In immutable, append-only storage, encrypt each person's records with their own key and destroy the key on request (crypto-shredding). State the assumption that the cipher stays strong for the whole retention period; key management is now the critical system.

Keep deletion markers long enough that consumers that were offline do not resurrect deleted keys.

Three copies in one rack are roughly one copy. Their value comes from separate failure domains, and the "eleven nines" figure assumes independence.

Replicas obey a bad migration within milliseconds, so you also need backups separated in time and authority (object lock, point-in-time recovery) and a monthly restore test.

Re-replicating 4 TB at 200 MB/s takes 20,000 s, about 5.6 hours: the window in which a second failure loses data.