Log truncation and retention policies

Why is log retention paid by the live path?

The question is not "when is a record old?" but "who still needs it?" For a database log, the parties are:

1. Recovery: everything since the oldest dirty page's recoveryLSN.
2. Undo / MVCC: the undo data for the oldest open read view.
3. Replicas: everything after the slowest replica's replay position.
4. Archive / PITR: everything since the oldest base image you intend to restore from.
5. Logical consumers / CDC (change data capture): everything after the slowest downstream reader.

The truncation point is the minimum over all five. So the log's size is controlled by the worst-behaved consumer among five subsystems that have nothing else in common, and any one of them alone can fill the disk. This is exactly why a forgotten Postgres replication slot takes down a primary: party 3 or 5 stops advancing, the minimum stops moving, and a healthy system runs out of disk for a reason that is invisible in any write-rate graph. (It is Nadia's problem.)

The design fork: who pays?

When a retainer falls behind, somebody has to break. The policy is a choice of whom.

All three are the same cost, relocated. There is no fourth option, and a system that seems to offer one has hidden the cost somewhere you have not looked yet. Tiered storage (old segments moved to cheap object storage) is the current favourite escape: it cuts the per-byte cost by about an order of magnitude and softens the policy question. The price is that the object store's availability becomes a dependency of your log, and historical reads are about an order of magnitude slower.

Keep by key: compaction

A log can answer two different questions: "What happened?" needs the history; "What is true now?" needs only the last record for each key. If every consumer asks only the second question, you can keep only the latest record per key. An unbounded log of size O(events) becomes a bounded one of size O(distinct keys).

The consequence is a change of kind, not just size: compaction changes what the log is. A time-retained log is a stream of changes. A compacted log is a table that happens to be stored as a log, and its durability promise narrows with it: it keeps every key's latest state, not the history. What that means for the consumers that read it (deletes as tombstones, rows that come back, the intermediate states that are gone) is in Compaction: the third mode.

Retention versus erasure

An append-only log with independent immutable backups is a system that structurally cannot forget. That collides with legal erasure obligations (for example, "delete this person's data"). The standard escape is crypto-shredding: encrypt each subject's data with its own key, and when erasure is required, destroy the key and let the ciphertext rot. This moves the problem into key management, and it quietly assumes that today's cipher will survive the whole retention period. State that assumption explicitly: it is a durability claim about cryptography, often made by people who are not cryptographers.

The pattern this article actually teaches

Four separate items in this mechanism (pinned undo, retained CoW snapshots, tombstone grace windows, replication slots) are the same thing:

Appearances:

1. MVCC history pinned by a long transaction (Postgres bloat, InnoDB history list).
2. A CoW snapshot pinning blocks and forcing allocate-on-write.
3. A Kafka consumer lag, or a Postgres slot, pinning log segments.
4. Cassandra gc_grace pinning tombstones.
5. LSM compaction blocked by an open iterator (a reader holding the old files).
6. Iceberg snapshot expiry blocked by a retained snapshot.
7. S3 versioning.
8. Git objects pinned by a reference.

Eight appearances across storage engines, streaming, table formats and version control. This is the governing rule of persistence and the single most reliable source of production incidents, because in every case the party that causes the cost is not the party that pays it, and no metric on the writer's side names the reader.

The practical corollary: every log-structured system contains a distributed liveness dependency that it does not advertise. Computing "oldest needed" requires every consumer to report its position. A consumer that neither advances nor disappears is worse than one that dies.

How checkpoints, compaction and truncation sit on three trade-offs