Snapshot-diff incrementality

How does a table format give you changes from metadata alone, and what limits that?

A table format is a layer of metadata that turns a directory of files into a table with atomic commits (Iceberg and Delta Lake are examples). Each commit creates a new snapshot: a metadata record listing exactly which files make up the table at that moment. Because the format commits snapshots atomically, it already knows which files each snapshot added and removed.

So the difference between two snapshots can be computed from metadata alone: no capture, no triggers, no log tailing. The table has become its own change stream, and the consumer's cursor is a snapshot id.

Two limits

1. Fidelity. A file-level difference gives you added and removed files, not rows. A copy-on-write update shows an entire rewritten file as changed. To find the one modified row, you need either row-level lineage metadata (an id per row that survives rewrites) or a re-read and compare.

2. Retention, which is operational. Maintenance jobs (snapshot expiration, vacuum) delete the history your consumer's cursor points into. Retention becomes a two-party negotiation between the maintenance job and the slowest reader. If the cleaner expires snapshots older than 7 days and your consumer is down for 9 days, its cursor points to a snapshot that no longer exists. This is Tariq's "snapshot not found".

This is structurally identical to the log-retention problem in Detect (log segments versus the CDC reader), one level up. It is the second appearance of the same shape.

In one line: a versioned table is a change log with a garbage collector, and your cursor is racing it.

Case, resolvedConstructed case · Tariq at Quillpost

A late batch the watermark never saw

An append-only events job missed GPS pings uploaded two days late, and a snapshot-diff reader failed with snapshot not found after maintenance expired its history. Where it started: Coarse incrementality →

Late GPS data. Noor adds a 3-day re-processing window. For corrections older than that, a documented manual re-run of the specific partition is the procedure. Both the window and its limit are written in the pipeline's readme: "corrections older than 3 days are not picked up automatically."
Snapshot not found. Tariq and the platform team agree: snapshot retention is 14 days, and consumers raise an alert after 3 days without progress. If a consumer falls behind by more than the retention, it does a full re-bootstrap instead of trying to resume. The retention number is now a documented contract, not a surprise.
A lesson. For events, no CDC infrastructure was ever needed. He asks of every future table: "is it append-only and partitioned by arrival?" before he asks about logs.