Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Compare-and-swap: the object store's only transaction primitive

How do multi-object transactions follow from one conditional write, and what does its latency set?

Here "atomicity over n items is atomicity over one item" stops being an abstraction.

What an object store offers. Cloud object storage (Amazon S3, Google Cloud Storage, Azure Blob) lets you store whole files ("objects") by name through HTTP requests. It has no in-place partial write, no multi-object atomicity, no service, a floor per request and a price per request. It offers essentially one useful thing for our purpose: a conditional write. Put-if-absent: write this key only if it does not exist yet. Or put-if-etag-matches: write only if the key still has the version (the "etag") I last read. One key, one atomic swap. This is (CAS) over HTTP.

From that one primitive, multi-object transactions follow mechanically:

  1. Write all new data objects under fresh unique names. This is safe without any coordination, because nothing refers to them yet. Immutability makes speculative writes free. (A dividend of immutable storage that appears only once you need transactions.)
  2. Build a new metadata object that describes the resulting state of the table (the list of files that make it up).
  3. Conditionally swap the pointer from the old metadata to the new metadata.
  4. If the swap fails, someone else committed first: re-read and retry. If you give up, the orphan files you wrote are the garbage collector's problem.

Every element is something you already know: optimistic concurrency control, an immutable , an indirection layer, a garbage collector. What is new is the cost structure.

The number that governs everything downstream

An object-store round trip takes roughly 10–50 ms at the median (p50) and much more at p99. A needs to read the current metadata, write new metadata, and do the conditional swap. So the ceiling on commits per second for a single table is on the order of tens per second. A database, bounded only by , reaches tens of thousands. That is three orders of magnitude. This one ratio explains:

  • why streaming writers batch their data into commits every minute or so;
  • why the small-files problem is structural, not an accident;
  • why concurrent writers to one table are rare and unpleasant.

The prediction came true

Before S3 offered conditional writes, that storage had no atomic primitive at all. The architectures built on top of it grew an external component whose sole job was to supply one. Iceberg requires a (a small service that stores "which metadata file is current") that is capable of an atomic pointer swap. Delta Lake on S3 needed a separate coordination service behind its log-store abstraction; on systems that already had put-if-absent it could use the filename itself as the CAS (two writers cannot both create version file number 00042). Same mechanism, different substrate:

Abort asymmetry, inverted

In a database, an aborted transaction wastes its own CPU time. In a lakehouse, an aborted commit may waste gigabytes of Parquet files (a columnar file format) that you already wrote. The exception is when the conflict class permits a : if the concurrent commit touched different files and your operation is a blind append, you keep your data files and retry only the metadata step. But a MERGE (an operation that reads existing rows and rewrites files) that read files that the other writer deleted cannot rebase, and the whole write is lost. So the practical question for any lakehouse operation is not "will it conflict?". It is "if it conflicts, can it be rebased?", because that decides whether a retry costs one round trip or the entire job.