Choose a serialization format
One system writes records and others read them: services exchanging messages, a pipeline writing files, an engine storing keys. A reader can only decode bytes if it knows the schema, so every format decides where that description lives and when the reader learns it. Separately, it decides whether data is laid out row by row or column by column. Which format fits who reads the data, when, and how?
You will end with where the schema lives, the data layout, and whether keys need byte-order encoding.
Before you choose
Ask who will read this in three years, and how often
That sets how late the reader may learn the schema. It is the only question here about people, and the one most often skipped. Data read once by its own writer and data read for years by teams you have not met need opposite answers.
Decide where the schema lives
Will every reader be rebuilt together with the writer?
True inside one service or one deploy. Across team or company boundaries it is almost never true, and assuming it is causes most of the damage.
Why
If reader and writer always change together, the schema can be fixed when the code is built: the cheapest option, with nothing on the wire.
Are records written in batches (files), or one at a time (messages)?
BatchesSchema in the file header
One at a timeGo to 3
Why
A batch can carry its schema once in a header or footer, so the question of a separate registry disappears. A single message cannot afford to embed the schema; it must refer to one.
Must a reader be able to skip fields it does not understand, without fetching the schema?
Why
Skipping an unknown field needs framing around each field. Without it, records spend zero bytes on description, but every reader depends on fetching the schema from somewhere.
Decide the layout
Is the data mostly scanned, or mostly fetched one record at a time?
ScannedGo to 5
FetchedRow-oriented
Why
Scans read a few columns of many rows; lookups read all columns of one row. The gap between layouts is structural, and no tuning closes it.
Is the bottleneck bytes moved, or CPU cycles spent?
Bytes: data read from storage or sent over a network. Cycles: data already in memory, passed between processes or engines.
Why
Parquet is compressed and encoded to minimize bytes on disk; Arrow is laid out to be processed in place, with no decoding. They are complements: one for storage, one for compute.
Where you can land
Compiled-in schema
Use it when every reader is rebuilt together with the writer.
What it costs: a change to the schema needs every reader redeployed; any reader outside your deploy breaks.
Schema in the file header
Use it when records are written in batches.
What it costs: a fixed overhead per file, which sets a minimum sensible file size; readers must resolve the writer's schema against their own.
Tagged fields
Use it when individual messages, and readers must skip fields they do not know.
Each field carries a small number identifying it (as in Protobuf).
What it costs: about one byte per field, and field numbers are permanent: never reuse one.
Schema registry, untagged records
Use it when individual messages, and readers can always fetch the schema.
Each message carries a small schema ID; the schema lives in a registry.
What it costs: the registry needs the same availability as the data itself. If it is lost, the data cannot be decoded.
Row-oriented
Use it when records are fetched one at a time.
What it costs: scans read every column.
Columnar file (Parquet-shaped)
Use it when data is scanned and the bottleneck is bytes.
What it costs: encoding and decoding work, and the file must be complete before it can be read.
Columnar memory (Arrow-shaped)
Use it when data is scanned in memory and the bottleneck is CPU.
Before you build it: measure the conversion between Parquet and Arrow before assuming it is free; it often dominates.
If a storage engine compares keys as bytes
Use order-preserving byte encoding
Engines that sort keys as raw bytes need encodings whose byte order matches value order. You then own the details: flip the sign bit of signed integers (two's complement puts negatives after positives), handle negative floats, escape and terminate strings so a prefix sorts before its extensions, and store a separate collation key for text sorted by language rules.