Where does the schema live?
Who interprets a file, the writer once or every reader forever, and what decides splittability?
A file is the extreme case of a boundary: zero back-channel, unbounded time gap, and a reader that often arrives with nothing but a range of bytes. So every file format answers one question, and all the differences between formats are answers to it:
There are only three places:
- Nowhere. The reader infers it (guesses from the data).
- In the bytes. The file carries it.
- Out of band. A registry or catalog that both sides agreed on in advance.
Every format is one of these three, plus a decision about where in the file a reader is allowed to start. Nothing else about a format matters at a boundary. (How the bytes carry a schema, and what that costs per field, is encoding's question.)
The parse tax
The second question depends on the first: how much work stands between the byte and the value? We call this the parse tax. It is paid once per read, by the consumer, forever.
| Family | Parse tax | Why |
|---|---|---|
| Text | Highest | Reader parses characters and guesses types |
| Self-describing binary | Lower | Types are known; values are decoded |
| Binary with schema in a catalog | Lower still | Both sides already agreed |
| Memory-layout formats | Zero | Agreement is total: the bytes are the in-memory values |
This is the idea of early versus late binding. Binding means fixing the meaning of something. Early binding fixes meaning in advance (before reading), which moves cost off the read path and onto the agreement. Late binding fixes meaning at read time and does the reverse. Early binding is the spine of this whole section.
Splittability
A third property follows from the first two and matters more than it looks. Splittability is whether a reader that starts at a random byte offset can find the next record boundary without reading from the beginning. It decides whether N workers can read one file, or only one worker can.
Real jobs have overhead, but the ratio shows what splittability buys.
The popular saying "gzip is not splittable" is imprecise in a useful way. Splittability is a framing property (can I find a record boundary?), not a compression property. A single gzip stream cannot be entered in the middle, because decompression needs the bytes before it. That part of the saying is true. But the codec itself is not the problem. The real issue is that one big compressed stream destroys the writer's ability to place recoverable boundaries. Formats that cut the file into independent blocks stay splittable under any compression codec. (Some tools also write gzip in independent blocks that can be indexed, which makes the file splittable. The general rule stays the same: framing decides, not the codec.)
This is the granularity mismatch again. The unit of storage is the file. The unit of parallel work is the task. The format is what connects them, or fails to.