Bytes become an output

Why is byte count a choice here, and when is compression worth its CPU cost?

Byte count is now an output

Everything below this mechanism in a storage system moved bytes without knowing what they meant. Disks, caches, the network, even data structures: all of them take the number of bytes as an input. They are given a size and they pay a cost for it.

This mechanism is where byte count becomes an output. It is the first place where you choose the mapping from values to bytes. So it can change every number that disks, caches and networks give you. A 4× win in encoding is a 4× reduction in disk operations, in the amount of data on the network, in cache space used, and in the number of requests that must run in parallel to reach a target speed. Nothing underneath can offer such a lever.

Two new constraints appear here that had no meaning before.

1. The reader and the writer are separated in time and space, and neither can check that the other exists. Below this mechanism the only party was the machine. Here there are two parties and a contract between them. A contract is an informational constraint, not a physical one. The bytes are written once, and they are read later by programs that may not have been written yet.

2. CPU time and bytes can be exchanged at a rate you can tune. Below this mechanism you got the bandwidth the hardware gave you. Here you can buy bandwidth with CPU cycles. This is the first place where the bottleneck often moves from I/O to CPU, and the first where "just add cores" is a valid answer to a storage problem.

One constraint almost stops mattering: physical location. Here the data is a logical stream of bytes, and where it lands on a disk is storage layout's problem. What takes its place is decode locality: how much of the stream you must touch to recover one value.

Now the core tension of this mechanism, stated once and repeated in three forms:

The three forms of the same trade are:

• schema registry versus self-describing records (Serialization);
• footer metadata versus inline tags (Serialization);
• trained dictionary versus embedded dictionary (Lightweight compression and General-purpose compression).

Same trade, three costumes.

---

Why this mechanism has exactly three stages

This mechanism covers three groups of techniques. They are not a convenient list. Each group exploits a different kind of redundancy (repeated or predictable information), and they must be used in a fixed order.

The order is forced, not a habit. Each stage removes the redundancy that the next stage is bad at, and each stage makes its output look more random to the stage after it. Suppose a column is first turned into a dictionary, then run-length encoded, then bit-packed (all explained in Lightweight compression). What remains has far less for a general compressor (an "LZ" compressor, General-purpose compression) to find. But what does remain, such as string repetition across columns and across pages, is exactly what LZ is good at and what the lightweight codecs cannot see. If you reverse the order, you destroy the type structure that the lightweight codecs need.

So "Parquet with ZSTD" is a three-stage pipeline, not one choice. Most tuning mistakes come from treating it as one choice.

A folklore correction: "Parquet compresses 10×". This sentence mixes four separate effects: projection (reading only some columns), sorting, dictionary and RLE on low-cardinality columns, and the general-purpose codec. The general-purpose codec is usually the smallest of the four. Report them separately, or you will tune the wrong one.

---

The governing quantity: when does compression pay?

Compression is not free. You save transfer time, but you spend decode time. When is the trade good?

Let:

• S = the uncompressed size of the data,
• B = the bandwidth of the pipe the data travels through (disk or network),
• r = the compression ratio,
• D = the decode throughput, measured in output bytes per second.

Reading uncompressed data takes S / B seconds. Reading compressed data takes S / (rB) seconds to transfer, plus S / D seconds to decode. Compression wins when

S/(rB) + S/D < S/B.

Divide by S and rearrange. First: 1/r + B/D < 1, so B/D < (r − 1)/r. Taking the inverse gives the rule:

This is the whole economics of encoding in one line. Look at what it says about the ratio r:

The threshold is nearly flat after about r = 4. A ratio above ~4× buys you almost nothing on the read path. Its value is in the storage bill and in cases where the pipe B is small. Everyone chases ratio. The derivation says: chase decode speed.

Now use the two regimes that matter in practice.

• Local NVMe disk, B ≈ 3–7 GB/s. At r = 4 you need D > 4–9 GB/s. ZSTD decodes at about 1 GB/s per core. On one core it loses, on four cores it wins. This is why the old defaults LZ4 and Snappy come from the local-disk era: they were the only codecs above the line on one core.
• Object storage, B ≈ 50–100 MB/s per connection. B is now 30–140× smaller. Almost every codec clears the bar by a wide margin, so the right default shifts to maximum ratio. This is why newer formats moved to ZSTD.

That shift is an economic accident, not a permanent rule. It reverses if you read from a local cache or over a 100 Gb Ethernet fabric. The number to remember is the ratio D / B, not the name of the codec.

A folklore correction: ZSTD level and decode speed

For ZSTD-family codecs, decode speed is roughly independent of the compression level. Level 19 is about 50–100 times slower to write than a low level, and not slower to read. The saying "level 19 is too slow" mixes up the two paths. You choose the level from your write-time CPU budget and from how many times the data will be read. Read latency is not the deciding factor.