Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Table formats and log formats

Why does a set of files need a table format, and what follows from a log having no ending?

Table formats as metadata protocols

A file format's guarantees stop at the file. A dataset is a set of files, and a set has no description of itself: the filesystem knows names, not meaning. Nothing in a file format gives you atomicity (all-or-nothing visibility), snapshot identity ("the table as of yesterday") or a schema for a collection. So a second format appears one level up. Its content is not data but a description of which files currently make up the table. Examples are Iceberg, Delta and Hudi.

This is a familiar shape: immutable storage plus mutable intent forces an indirection layer and a collector.

  • The files are immutable (never changed in place).
  • But what the table means changes: rows are added, updated, deleted.
  • So you need an indirection layer (the says which files count now) and a collector (a background job that removes files nobody references).

The same shape appears in log-structured storage engines (manifests of sorted files), in multi-version databases (chains of row versions) and in copy-on-write object stores. How the description is laid out as a manifest tree, and how one pointer swap commits it, are metadata's questions; how concurrent writers conflict is a transaction's. Here the question is only what the engines must agree on.

Why this is a boundary topic

The metadata is specified, not internal. Several independent engines read and write it without sharing any code. That makes the metadata log a wire protocol with a compatibility story, version negotiation, and all the consequences of Principle A. It also explains why these formats standardize slowly and painfully: a format read by one engine can change freely; a format read by six is a treaty. The competition between them is being decided by adoption, not by technical merit (Principle B, playing out in public).

In one sentence: a table format exists because a set of files cannot describe itself, and it is hard because the description must be readable by engines that will never meet.

Line protocols and log formats

This is the inverse of columnar, derived from the inverse constraint: the file has no end. You cannot write a for something that is still growing. Therefore:

  1. No global index.
  2. Therefore no random access.
  3. Therefore the only operations are append and scan.

Everything else follows:

  • Each record must be self-delimiting, because a reader can attach at any moment and must find its place again (resynchronize).
  • The schema is per record, not per file, because a file-level declaration would have to be written before the records it describes and could never be amended.
  • Compression, if used, must be per block.
  • The identity of a position is a byte or sequence number. This is exactly the durable state that integration patterns need.

This is not a separate family of formats. It is the correct answer to unbounded, continuously written data, which is why it looks the same everywhere: ingestion protocols, access logs, and the log-structured layouts inside storage engines. (Where a log keeps each reader's progress, and what that costs, is data movement's queue-versus-log choice.)

In one sentence: . Everything else about log formats is that one fact restated.