Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

LIST cost and manifests

Why did table formats turn listings into files, and where did the cost move?

LIST cost → manifests

Derive the failure from first principles. A paginated listing returns about 1,000 keys per call, and each call needs the cursor returned by the previous one. So the calls are sequential. This is the one request pattern that cannot be run in parallel.

There are two escapes.

  1. and read it in one request. This converts O(N) sequential requests into one request plus O(N) bytes.
  2. Keep a service that holds the listing in memory. This brings back a coordination point and a scaling limit.

Everyone converged on the file. The reason was overdetermined: object stores also give no atomic directory operations (you cannot rename or replace many objects at once). A (a file that lists which data files make up a table) plus a compare-and-swap on one pointer key solves both the listing cost and the commit problem with one structure. Two independent pressures led to one design. That is why three table formats (Iceberg, Delta, Hudi) differ in almost every detail and agree exactly here.

The cost did not disappear. It moved. Manifests accumulate, reading them becomes the new listing problem, and so metadata compaction and snapshot expiry appear to service them: an immutable substrate grew an indirection layer, and the indirection layer grew a garbage collector, here over a billing model rather than a device.