Row binary formats: Avro, Protobuf, Thrift
What does agreeing on a binary row format buy, and where must the agreement live?
Row binary: Avro, Protobuf, Thrift, MessagePack
Now the constraint relaxes by exactly one step: the two parties can agree on something in advance. Immediately the data stops being human-readable, because readability was only there to stand in for agreement. Values become length-prefixed (the length comes first, then the bytes) or typed, rather than separated by delimiters. That kills escaping and restores framing for free: the reader knows how many bytes to take, so a comma or a newline inside a value is just a byte.
Where does the agreed schema sit?
This is the real split inside the family, and each choice has its own failure.
Schema carried in the file. The file is self-contained: a reader needs nothing external. The cost is per-file overhead, which sets a minimum sensible file size. This is the fixed-cost rule: every operation has a fixed cost, so there is a natural batch size, here "do not write a thousand tiny files". It suits durable data at rest, because the file may outlive every system that knew about it.
Schema kept out of band, with only an identifier in the payload. Messages become very small, which matters when the same schema would repeat ten billion times on a wire. The cost: the registry is now on the critical path for readability, and a lost registry means lost data. The cost did not disappear; it moved into a new dependency (encoding prices each escape).
The registry saves 400× the bytes, and in exchange, if the registry is lost, 10 billion messages cannot be decoded.
Field identity: number or name
A quiet decision with a permanent effect:
| Binding | Rename a field | Reorder fields | Danger |
|---|---|---|---|
| By number (Protobuf, Thrift) | Free | Free | Reusing a number is catastrophic, and the reader cannot tell field 7 now means something new |
| By name | Breaks readers | Free | Names must be unique and stable |
Neither is safer. They change which mistake can be detected. (How schemas may evolve is a separate question; what matters here is that the format chose the binding, and the choice cannot be undone once files exist.)
In one sentence: agreement buys framing and compactness; the only question left is whether the agreement travels with the data or is kept somewhere you must not lose.