Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Columnar files, Arrow and schema resolution

Why is Arrow not a competitor to Parquet?

Columnar file encodings

People usually say a columnar file is good because of its . That is imprecise. There are three separate wins, and you can measure them one by one, because they have different sizes in different situations.

1. Projection. Bytes read scale with (columns touched ÷ columns in the table). On a 200-column table where the query touches 3 columns, this alone is about 200 ÷ 3 ≈ 67×. On a 5-column table where the query touches all 5, it is 1×. It depends only on table width and query shape.

2. Encoding homogeneity. A column chunk has one type and one distribution. So type-aware encodings apply, and LZ matching is not interrupted by fields of other types. This is typically 2–4×, and it applies even when you read every column. That is why a columnar file beats a row file even on a narrow table where the argument predicts a tie.

3. Skipping. Per-chunk statistics (such as minimum and maximum) let you avoid decoding some chunks entirely. How well this works depends almost completely on clustering (Lightweight compression).

And there is a structural loss: to rebuild one full row you must touch every column chunk. is poor at for a reason no tuning can fix. The file's is a column chunk, the query's granularity is a row, and the mismatch causes .

Nested and repeated data: definition and repetition levels

Columnar layout assumes a fixed number of values per row per column. Repeated and optional fields break this. For the path user.phones[].number, a row may have zero, one or nine values. If you flatten everything into one buffer, you lose (a) which row each value belongs to, and (b) the difference between absent and empty.

The method from Google's paper uses two small integers per value.

  • The counts how many of the optional or repeated ancestors along the path are actually present. If it is lower than the maximum, the value is absent, and the level itself says at which depth it went missing: was user null, was phones an empty list, or was number null inside a present phone? A simple null bit cannot make this distinction. That is why nested columnar formats need something richer than a null bit.
  • The says at which repeated ancestor this value started a new list. Zero means "a new record starts here".
Three nulls, told apart by d
column user.phones[].number · max r 1 · max d 3

The elegant point is the bound: both levels are limited by the depth of the schema, not by the size of the data. The typical maximum is 3 or 4, so each level fits in 2 or 3 bits. Both sequences are highly repetitive, so reduces them to almost nothing. An arbitrary tree costs a fraction of a byte per value.

The cost is reconstruction: assembling full records needs a state machine that runs over all the columns in step. Projection queries are cheap. Retrieving whole documents is not.

Parquet has a hierarchy that is best read as three different fixed-cost decisions:

  • the (about 128 MB): the unit of parallelism; it must fit in the writer's memory;
  • the page (about 1 MB): the unit of , and therefore the unit of decode granularity;
  • the : holds the schema and the statistics.

The footer is at the end of the file. This has a consequence that people feel but blame on the wrong cause. On you must read the tail of the file to learn the footer's length, then read the footer, then read data. That is two round trips of 30–70 ms before the first data byte, often reduced to one by a speculative read of the tail. For a 1 GB file this is noise. For ten thousand files of 100 KB it is the entire job. (Check: 10,000 × 2 × 50 ms = 1,000 s of waiting, if run one after another.) The small-files problem is partly a decision. It is not only a metadata-service problem, and no catalog tuning removes these round trips.

Arrow, and why it is not a competitor to Parquet

(Apache is a standard in-memory column format.) Different goals give different designs, and asking "which is better?" is a category error. The interesting question is what the conversion between them costs. Almost nobody measures it.

Arrow's constraints, and what each one forces:

A value must be usable by the CPU with no decode step. So fixed-width values sit at offset i × width. No , no . Random access is O(1) (constant time), and arithmetic can use SIMD. The price: Arrow data in memory is often 2–10× larger than the Parquet file it came from. That is not a defect. It is the space-versus-time trade taken to the opposite end.

Nulls must not cost a branch. So Arrow uses a separate (one bit per value saying "present or not") instead of or a nullable union type. Let us derive the cost. One bit on an int64 column is 1 ÷ 64 = 1.56% extra space. It removes a data-dependent branch whose misprediction costs about 15 cycles, while the arithmetic itself costs about 1 cycle. You compute over all values without checking, and apply the mask afterwards. Counting nulls becomes popcount (a CPU instruction that counts 1-bits), 64 values per instruction. You could pay eight times that space and still come out ahead. A sentinel value, by contrast, takes one value away from the domain and brings the branch back. That is why designs that use sentinel nulls keep losing.

Variable-length data needs O(1) access. So Arrow uses an : element i occupies the bytes from offsets[i] to offsets[i+1]. Length is derived instead of stored, so one buffer is enough instead of two. The price: 4 or 8 bytes per element even for empty strings, no appends out of order, and a 32-bit offset limit that forces the "Large" variants once data passes 2 GB.

SIMD must apply cleanly. So buffers are 64-byte aligned and padded to multiples of 64 bytes: the size of a cache line and the width of an AVX-512 register. Here this mechanism reaches straight down to the hardware and lets hardware geometry decide a wire format.

Folklore to correct: "zero-copy" means no serialization step, not no transfer. Arrow Flight (a network protocol for Arrow data) still moves bytes over a network like anything else. What disappears is the encode and decode CPU at both ends. In a typical path from Parquet to a dataframe through a row-oriented database driver, that CPU is 60–80% of total time. Quote the saving you really get, not the one the phrase suggests.

Schema fingerprinting and writer-reader resolution

Bytes were written under schema W (the writer's). The reader wants them as schema R. Resolution is a function of this pair. 's version of resolution works like this:

  • fields are matched by name;
  • a field missing in the data is filled from R's default value;
  • a field present only in W is discarded;
  • widening promotions are allowed: int → long → float → double.

This needs both schemas at read time, no exception, because of the zero-framing decision made earlier. So W must be recoverable: either embedded in a file header (where it costs nothing per record) or referenced by a few bytes (where it cannot be embedded).

Fingerprinting is applied to a schema. The hash is computed over a : whitespace, the order of fields in the JSON text, documentation strings and aliases are removed. Identity must be semantic, not textual. (Two schemas that differ only in a comment must have the same .)

Collision check. For n schemas and a 64-bit fingerprint, the birthday bound puts the chance of any collision at about n² ÷ 2^65: about 2^−32 at roughly 92,000 schemas, and about 0.5 only at four billion. No real organization registers anywhere near that many schemas, so collisions are not the part that breaks.

What does break is subtler:

  • Backward compatible: the new reader can read data from old writers.
  • Forward compatible: the old reader can read data from new writers.

A registry set to be non-transitive checks each new version only against the previous version. A chain of individually valid changes can then be invalid across two steps. Example: version 2 adds a field with a default; version 3 removes a different field. A consumer pinned at version 1 can now fail on a topic that the registry approved at every step. Who pays? Whoever replays from the start of the topic, six months later, in another team. (This is Nikhil's first incident.)

And now the deeper limit, which is the real lesson of this article:

Type compatibility can be checked by a machine. Meaning compatibility cannot: a field that changes units (grams to kilograms: Nikhil's second incident) passes every compatibility check ever written, and so does an enum that gains a member. Why every structural check is blind to meaning, and what to do instead, is schema evolution's hard boundary.