Serialization: where the description lives

How do we turn a record into bytes, and who tells the reader what the bytes mean?

What is the reader told, and when?

Compression asks how to make bytes fewer. Serialization asks something that comes first: what is the reader told, and when is it told? These are different questions. Mixing them up is why most arguments about file formats lead nowhere.

Start with what a reader minimally needs to turn bytes into values. For each value it needs three things:

• its extent: where it starts and where it stops;
• its type: how to interpret the bytes (a 32-bit integer? a string?);
• its identity: which field this is.

This triple must come from somewhere. There are exactly three places:

1. the byte stream itself (a self-describing format such as JSON);
2. an artifact fetched at read time (the schema is written elsewhere, for example in a registry, and the reader looks it up);
3. the reader's own code (the types are compiled into the reader: compile-time binding).

There is no fourth source. Every format is a position somewhere in this three-way split, usually a mixture, and the mixture decides when your failure appears. This is the old software question of early binding versus late binding. The serialization part of this mechanism is that question, not just an example of it.

---

What the density dial really controls

Here is the derivation that organizes the rest of this article.

A reader can skip a field it does not understand if and only if it can find the field's extent from the stream. Finding extent needs either a length prefix or a type tag with a known width. Call this the framing cost. It is paid per field, per record, forever.

Protobuf pays it. Each field starts with a tag. The tag is a varint (a variable-length integer, explained below) that holds (field_number << 3) | wire_type. The low 3 bits say which kind of value follows (a number, a length-prefixed block, and so on), and these kinds are all ones whose length can be determined. The remaining bits are the field number. For field numbers 1–15 the whole tag fits in one byte. That one byte per field is exactly the minimum price of skipability. It is why a Protobuf consumer built from an old .proto file can pass over fields it has never heard of.

Avro pays zero. An Avro record is simply the field values written one after another: no tags, no lengths on fixed-width types, no separators. This means an Avro reader cannot find field 7 without having already parsed fields 1–6 using the writer's exact schema. It cannot skip an unknown field, because it cannot find where the field ends.

The consequence:

Density and self-sufficiency are one dial. The cost did not go away. It moved from bytes into an availability dependency (a service that must be up and correct whenever data is read).

How big are the formats, really?

Folklore exaggerates the size differences. Take one record with ten fields, a mix of small integers and short strings. It has about 40 bytes of real payload, and field names average ten characters.

Now put a thousand such records into one block and run ZSTD over it. The field names are byte-for-byte identical in every record, so LZ matching removes nearly all of them. JSON compresses 8–12×. Avro compresses only 1.5–2.5×, because it had little redundancy left to find. After compression JSON is roughly 1.3–2× the size of Avro, not 9×.

Worked arithmetic. 190 B ÷ 10 (a middle ratio) ≈ 19 B for JSON; 22 B ÷ 2 = 11 B for Avro. Compressed JSON is about 1.7× Avro.

So the size argument against self-describing formats is mostly false once compression is in the pipeline. Stop using it. What remains of the argument:

• Parse CPU. Conventional JSON parsers run at 300 MB/s to 1 GB/s. SIMD-based parsers such as simdjson reach about 2–4 GB/s. Avro and Protobuf decode at 1–3 GB/s with far fewer memory allocations.
• No random access inside a JSON document without scanning.
• Ambiguous number types. JSON numbers are decimal text. Above 2^53 a JavaScript number cannot hold every integer exactly, so a 64-bit ID written by a JVM producer and read by JavaScript can be silently changed. This is a live source of silent corruption.
• Above all: no contract. Every assumption about a field is checked at read time by a consumer who did not write it. Schema-on-read does not remove the coupling. It removes the record of the coupling: the consumer still depends on field names and meanings, and that dependency now lives in unversioned application code where nothing can check it.

---

The impossibility at the centre of serialization

Three properties are wanted. You may have two.

• Dense: almost no per-field overhead.
• Self-sufficient: can be decoded with no external artifact.
• Skippable: an unknown field can still be walked over.

Where the main formats stand:

The three escapes and what each costs

1. Amortize the description over a batch. A file header or footer makes the schema free per record and restores self-sufficiency. This is why the Avro file format needs no registry and the Avro message format does. The cost: it works only when records are batched, so it is not available for single-message streaming; and it creates one point whose corruption destroys the whole file. This is also the idea that a fixed cost sets a natural batch size: the fixed cost here is a schema document, so the natural batch is "one file".

2. Content-address the description. Write a 4–5 byte fingerprint (a hash of the schema) in front of each message and look the schema up from it. The cost: a read-time dependency on a service, and an unlimited retention duty. Every schema ever registered must resolve forever, because someone will replay from offset zero.

3. Spend a fixed tag per field. One byte buys skipability for good. The cost: field numbers become an immutable namespace. You can never reuse a retired field number, because bytes written years ago still carry it. If you reuse it, an old record silently decodes into a new field that has a compatible wire type. Teams usually discover this failure after it has happened.

Unknown fields are a policy, not a detail. Fail-fast versus fail-degraded is decided here: Protobuf silently keeps or drops unknown fields, Avro fails hard without the writer's schema, JSON parsers vary with library settings. Strict versus permissive parsing, and quarantine versus stop-the-line on a malformed record, are all serialization policy. They decide whether a bad deploy upstream causes an alert or a slow build-up of wrong numbers.