Encoding, Serialization & Compression
What do we turn our values into when we turn them into bytes, and what does that choice cost?
Who this mechanism is for: a developer who writes code, sends JSON over HTTP, maybe uses Kafka or Parquet files, but has never studied how values become bytes, why one file format is ten times smaller than another, or what "compression" really does. You do not need any database or systems knowledge. Every technical term is explained the first time it appears, and again in the glossary at the end.
The story we will follow
Nikhil is a backend engineer at Tradewind, a small logistics start-up that tracks parcels. Warehouse scanners create a scan event each time a parcel is scanned: which parcel, which scanner, the time (in microseconds), a status such as LOADED or DELIVERED, the weight in grams, a latitude and longitude, the depot name, and an optional list of handling notes. Tradewind receives about 400 million of these events per month. Each event carries only about 40 bytes of real information. The events travel through a message queue (a Kafka topic called scans), are written in batches as Parquet files into cloud object storage, and are then queried by dashboards and analysts. The data is kept for three years.
Nikhil owns this pipeline. Sofia is the data platform lead. She reviews formats, owns the storage bill and asks the questions that Nikhil would prefer to have asked himself. During one busy quarter Nikhil will meet five problems, one at the start of each article below. Each problem looks like a bug. Each one is actually a consequence of how values are mapped to bytes. At the end of each article we go back and solve it.
The main idea of this whole mechanism:
How this mechanism is organized
- 1
Bytes become an output
Why is *byte count* a choice here, and when is compression worth its CPU cost?
CaseBetter compression, slower scans
- 2
Serialization
How do we turn a record into bytes, and who tells the reader what the bytes mean?
CaseThe replay that crashed and the weight off by 1,000
- 3
Lightweight compression
How do we shrink a column of values while it stays usable (comparable, searchable, summable)?
CaseSame data, files three times larger
- 4
General-purpose compression
What does the final "zip-like" pass do, and what does it destroy?
Case256 KB read for a 100-byte answer
- 5
What encoding forbids
What can this mechanism never give you at the same time, and what does it hand to whatever reads its bytes?
CaseA one-page rule for every new dataset
After that: a summary, hands-on labs, interview questions and a glossary.
One request: do not skip the numbers. This mechanism gives many sizes and speeds ("22 bytes", "0.3–4 GB/s"). The ratios between them are the real content.
What do I do, and what does each option cost?
Choose a serialization format
Data is written by one system and read by others, possibly years later: where does the schema live (compiled in, in a file header, per field, or in a registry), and is the data laid out by row, as a columnar file, or as columnar memory?
ChooseChoose lightweight encodings
Choosing how each column of a columnar file is encoded: decide first whether it must stay queryable, then row order, then dictionary, run-length, delta, frame-of-reference and bit-packing by what the values look like.
DesignChoose general-purpose compression
Whether to put a general-purpose codec (ZSTD, LZ4) on top of your data at all, and if so with what block size, level, dictionary, checksum and order relative to encryption.
Design
Can I see it myself?
Find your own break-even point D/B
Compress a 300 MB JSON log at zstd levels 1 to 19 and check, against a local disk and an object store, whether decoding beats reading.
MeasurePython with zstandardHow big is JSON really, after compression?
Encode 1,000 parcel-scan records as JSON and as Avro, raw and after zstd, with random and with sequential IDs.
MeasurePython with fastavro, zstandardMake `memcmp` sort your numbers correctly
Sort 10,000 integers and floats by their byte encodings, with and without the sign-bit flip and the float rule, and test composite-key escaping.
MeasurePythonVarint and zigzag: where the bet wins and loses
Bytes per value under varint and zigzag + varint for small counts, IDs, random 64-bit values and small signed numbers.
MeasurePythonRow order versus codec
Write 2 million rows to Parquet shuffled and sorted under NONE, SNAPPY and ZSTD; compare sizes and the depot column's row-group statistics.
MeasurePython with PyArrow, NumPyOne outlier sets the width
Bit-pack a block of 128 values, add one outlier, then patch it out with PFOR.
MeasurePython with NumPyEncoding by hand: dictionary, RLE and delta
A 10-million-value country column and a sorted timestamp column: dictionary, RLE before and after sorting, delta, and gzip, computed by hand.
MeasurePython with NumPyAre your floats decimals in disguise?
A million latitude-like doubles with six decimals: zlib on the raw bytes against scaling to integers and frame-of-reference.
MeasurePython with NumPyBlock size, ratio and point-lookup cost
Compress 200,000 JSON rows in blocks from 1 KB to 1 MB; compare ratio and the cost of decoding one block for a point lookup.
MeasurePython with zstandardDoes a trained dictionary rescue small messages?
Compress 100-byte JSON messages one at a time with and without a trained zstd dictionary, then decode with the wrong one.
MeasurePython with zstandard
Can I reason about it under questioning?
Interview questions
Short questions on encoding and compression, such as serialization formats, sort keys, columnar files, lightweight encodings and codecs, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on encoding and compression, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Read-optimized vs. write-optimized
For example: B-tree vs. LSM, merge-on-read vs. copy-on-write.
Space vs. time
For example: compression, indexing, materialization, caching.
Ordering vs. parallelism
Global order costs serialization; parallelism costs order.
Fail-fast vs. fail-degraded
Reject or continue when something is wrong.
Precompute vs. compute-on-read
For example: materialized views, ELT vs. ETL, schema-on-read.
Show all 12 trade-offsShow fewer
Coupling vs. coordination cost
For example: 2PC vs. sagas, schema contracts vs. schema-on-read.
Mutability vs. immutability
In-place update vs. append-only + GC; buys reproducibility and cheap concurrency, pays in space and janitorial work.
Cost as a substitutable resource
Money buys position on every other axis; the only axis denominated in a different unit.
Early binding vs. late binding
When structure is committed; determines where the failure surfaces, at write time or read time.
Latency vs. throughput
For example: batching, group commit, micro-batch.
Generality vs. specialization
For example: general engines vs. purpose-built structures.
Simplicity vs. adaptivity
For example: static plans vs. adaptive execution.
Ideas this mechanism meets that keep coming back, mechanism after mechanism.