The commit point

How does one pointer swap make a table commit atomic, and what ceiling does it set?

Object stores give no transaction across several objects. But consider this: if the whole visible state of a table is reachable from exactly one pointer, then making any number of file additions and removals visible at once reduces to swapping that one pointer.

Every commit therefore has two steps:

1. Write new immutable metadata. Readers cannot see it, because nothing points to it yet.
2. Compare-and-swap the pointer from the old root to the new root. Everything becomes visible at once.

def commit(catalog, table, base_root, new_files, removed_files):
    new_root = write_new_metadata(base_root, new_files, removed_files)  # step 1: invisible
    ok = catalog.compare_and_swap(table, expected=base_root, new=new_root)  # step 2
    if not ok:                      # someone else committed first
        latest = catalog.current(table)
        if conflicts(latest, new_files, removed_files):
            raise ConflictError     # real conflict: give up or redo the work
        return commit(catalog, table, latest, new_files, removed_files)  # retry on top of latest

Concurrency falls out of the CAS. Two writers race. One swap fails because the pointer moved. That writer re-reads the table state, re-checks its changes against the new state, and retries. This is optimistic concurrency control (OCC): do the work without locking, and validate at commit time, with the catalog as the validator. (OCC is also used in row-oriented databases. Here it is the foundation of table formats.)

The consequences are sharper than people expect:

• The catalog is the linearization point of the table. (Linearization point: the one instant at which an operation takes effect for everybody.) It is not a side system next to the table. It defines what "the table" means at any moment. So catalog availability equals table writability, and the catalog's own consistency model sets a ceiling on the table's isolation level. You cannot get serializable tables (behaviour as if writers ran one after another) from an eventually-consistent pointer store.
• Write throughput has a hard ceiling of one successful commit per catalog round trip, however much data each commit carries. Worked example: if a commit's round trip to the catalog takes 100 ms, the table accepts at most 1 / 0.1 = 10 commits per second. A stream of 50,000 rows per second therefore must be batched: at 10 commits per second that is 5,000 rows per commit at the very most, and in practice one commit every few seconds. That is why streaming writes into tables are micro-batched rather than record by record. The commit round trip is the fixed cost, and the batch size follows directly from it. (Batching raises throughput but delays the first row: the latency/throughput trade-off.)
• A catalog without an atomic CAS cannot support concurrent writers at all. Single-writer, file-system catalogs are not a simplification; they are a consequence. This is the real reason catalogs moved from "a file in the bucket" to a service with a conditional-update API.

Unsettled question: should catalogs stay thin (only hold the pointer) or grow fat (also do planning, pruning and access control)? The thin design keeps the linearization point small and fast. The fat design lets the catalog prune before the engine sees anything, which saves cost. Both are being built now. The answer is open.

Atomicity over an object store is one pointer swap. So the catalog is not a side system. It is the table's linearization point.