Optimistic commit on a metadata log
Why are concurrent lakehouse writers a different regime from database writers?
A lakehouse table's state is a numbered sequence of atomic versions: Delta's ordered JSON entries with periodic checkpoints, or Iceberg's tree of snapshot metadata over manifest lists over manifests. A commit reads version N, computes its changes, and tries to publish version N+1 atomically. Conflict detection happens in the gap between the read and the attempt.
The granularity question is the crux. Detection works on files and partition predicates, not on rows. Delta compares the files you read, and the predicates you used, with what concurrent commits added or removed. Iceberg checks whether files matching your predicate appeared since your read snapshot, with a configurable serializable or snapshot behaviour for row-level operations.
Now put that into the conflict formula from Why concurrency is different (expected conflicts grow like L²k²/2N), but with N counted in files instead of rows:
| Database | Lakehouse table | |
|---|---|---|
| Conflict unit | Row | File / partition |
| Typical N | 10⁶ rows | 10⁴ files |
| Typical k per writer | 10 rows | 100 files |
| k²/N per pair | 10²/10⁶ = ~10⁻⁴ | 100²/10⁴ = ~1 |
| Commit latency (sets W) | ~100 µs | ~50 ms |
Both factors move the wrong way at once, and they multiply. The per-pair conflict probability is three to four orders of magnitude higher. And W, which enters the formula squared (because L = λW), is about three orders of magnitude larger. Concurrent multi-writer lakehouse tables are not a slightly worse version of concurrent database writers. They are a different regime, six or more orders of magnitude apart in conflict pressure.
So the usual operational advice is arithmetic, not laziness:
- One writer per table, or writers partitioned so that they never touch the same files (k/N goes to zero by construction).
- Maintenance jobs separated in time from ingestion. Compaction rewrites files by definition, so it conflicts with everything by definition. Compaction is the transaction that cannot be made disjoint.
(This is the second problem of Tomas: the two nightly jobs overlapped, and the loser lost 40 minutes of work.)
Two more mechanisms follow from the log's own growth:
- To read the current state you read the last checkpoint plus every commit since. The read cost grows linearly with the number of commits. Hence the table format writes a checkpoint every N commits. This is the same checkpointing that databases use to bound recovery time, reappearing for the same reason: bound the cost of rebuilding state from a log.
- Data files plus a metadata indirection plus vacuum is the "immutable storage grows an indirection layer plus a collector" pattern for the fifth time, now at table granularity.
An honest note on claims. Whether a given table format's "serializable" mode is truly serializable depends on the validation it performs against predicates. The guarantees differ between formats, and between operation types within one format. This is a place where the documentation is more confident than the mechanism justifies. The reliable move is to reason from what the validation actually checks, not from the name of the level.