Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Columnar files and in-memory formats

Why is a columnar file readable only when complete, and what does zero-copy really decide?

Columnar files: Parquet and ORC

Column layout stores each column's values together. This makes scans over a few columns cheap, because a reader loads only the columns it needs and each 64-byte is full of useful values. What is new at a boundary is that a columnar file is a different kind of object from a row file: it is a small database, and it must publish an interface to its own inside.

  1. In column-major order, the values of one record are scattered across the file.
  2. A reader that wants 3 of 200 columns must jump directly to certain regions.
  3. Direct jumping needs a directory of .
  4. A directory for the whole file can only be written when the writer knows all the offsets, which is only at the end.
  5. So the metadata goes in a footer (at the end of the file).
  6. So the file must be finished before it can be read.
  7. So are structurally unable to be tailed or streamed.

This chain explains why nobody writes a live stream straight into a columnar file, and why "convert to columnar" is always a separate, batched step later.

What that directory buys a reader (projection, homogeneous encoding, skipping) is encoding's arithmetic; here the point is the price: an ending.

Because the writer summarizes what it wrote anyway, it can cheaply publish statistics (for example, the minimum and maximum value of each column in each block). Then an engine that never talked to the writer can skip whole blocks: if the query asks for weight > 900 and a block's maximum is 500, skip it.

So the file now exports a query interface, not only bytes. A file format has crossed the line from representation into protocol.

In one sentence: columnar files are readable only when complete, because the price of random access into a file is a , and .

In-memory and IPC formats: Arrow and FlatBuffers

Push early to its limit and the falls to zero. Define the layout so that the bytes on the wire are the bytes in memory. Reading becomes pointer arithmetic: nothing is decoded; the buffer is mapped and used. ( means inter-process communication: moving data between programs.)

What it costs: total agreement. Both sides must accept an identical physical representation: integer width, alignment, how are encoded, the order of buffers. There is no negotiation and no variation, no "my version encodes decimals differently". This is acceptable because the time gap is almost zero and both parties usually run at the same time. It is a poor bargain for a file that must be read in 2033, and these formats do not claim otherwise.

Two consequences:

  1. The fixed cost has moved here. When storage and network stop being the main cost, becomes the bottleneck. This format removes it. This is the same pattern as in hardware: as one part gets faster, the fixed cost moves to the next one up, and a new mechanism is invented to attack it there. (What saves in CPU is encoding's measure.)
  2. A shared representation across processes and languages turns an N-by-N conversion problem (every system converting to every other) into N adapters (each system converting to the one shared form). It is a coordination win of the same kind as in Principle B, not a speed win.

In one sentence: It is the decision to pay the whole cost as agreement instead of paying it per read.