Block size, level and trained dictionaries

Who pays for each choice of block size and level?

Trained dictionaries: the most underused tool of the last pass

Follow the small-input problem to its end. A 200-byte JSON message compresses by only about 1.1× with any codec. The reason is worse than "the window is empty." With dynamic Huffman coding, the code table that is sent along with the message is larger than the message. The description of the model is bigger than the data it describes.

This should sound familiar. It is the central tension of this whole mechanism: JSON's field names compete with its values, Avro's schema competes with its records, and now the entropy table competes with the message. The resolution is the same each time: move the description out of band into a shared artifact, and accept the same bill.

A ZSTD trained dictionary is about 110 KB, built by sampling representative records. It holds two things:

raw content that pre-fills the LZ window, so even the first byte of a 200-byte message can match against it;
pre-computed entropy tables, so no table needs to be sent.

Typical result on small structured records: 2–5× where compression without a dictionary gained nothing.

The cost is the one you already know. You now have a versioned artifact that every writer and every reader must have, forever, for every message ever written. A missing dictionary does not give a type error. It gives unreadable bytes. The duty to keep the dictionary never ends, distribution is your problem, and a version mismatch is a data-loss event, not a compatibility warning.

The trained dictionary is the schema registry again, after the Avro registry and the shared value dictionary. Three independent mechanisms, at three levels of abstraction, arrive at the same answer: "put the description somewhere else and accept a coordination dependency."

Brotli

Brotli has three differences from ZSTD, each with a consequence.

A static dictionary of about 13,000 common web strings, plus transforms, built into the codec: HTML tags, HTTP header names, frequent English words. This is a trained dictionary whose distribution problem was solved by putting it in every browser. It is why Brotli wins clearly on small web payloads and gains nothing on binary data.
Context modelling for literals: it uses the previous bytes, grouped into contexts. This is a cheap approximation of the higher-order models above, and is why Brotli edges out ZSTD on text at similar levels.
Larger windows at high levels.

Its decompression runs at 300–400 MB/s against ZSTD's 1–1.5 GB/s. That confines it to its niche: HTTP content-encoding, text, and the fact that browsers already include a decoder. Outside that niche, ZSTD generally wins.

The transferable lesson: a static dictionary is a bet on a domain, and it pays in proportion to how right the bet is. Brotli is a specialized codec dressed as a general-purpose one. The pattern recurs: a system tuned so hard for one workload that its benchmark wins do not carry over to others.

Level and block size: the two dials

These two dials are where practitioners spend most of their decisions, so they deserve the most attention.

Level controls the encoder's search, and nothing else

From the table above: going from ZSTD level 1 to level 19 costs 50–100× more compression time to buy about 25% more ratio, and it costs essentially nothing at read time. Two conclusions follow. The second one is the one people get wrong.

The level is chosen by the write path's budget, not the read path's.
"We use a low compression level to keep queries fast" is a category error. It makes queries slower, because more bytes are left to move, while it saves CPU on a path that the reader never uses.

The right level depends on the read-to-write ratio and on where the data is now. An ingest pipeline that writes 100 MB/s per core cannot afford level 19 at 5–10 MB/s. A nightly compaction job (a job that rewrites many small files into fewer big ones) that rewrites data which will then be read thousands of times can afford almost anything. This gives a pattern that keeps being reinvented: write hot data at level 1–3, and rewrite it at level 9–19 during compaction. Tiered compression and tiered storage are the same idea. As data cools, its read-to-write ratio rises, so effort spent on it pays back better. The tiering policy you already need for storage class is the policy you need for level. Keeping them in separate configurations is how they drift apart.

The diminishing returns are structural. The match finder searches for matches that are progressively rarer, shorter or further away. Each extra match saves fewer bytes and costs more search. Ratio grows roughly with the logarithm of effort. Now combine this with the break-even from Bytes become an output, D/B > r/(r−1), whose threshold is nearly flat after r ≈ 4. The supply curve for ratio flattens, and the demand curve for ratio flattens. This is the quantitative reason why almost everyone belongs at level 1–3 and almost nobody needs level 19.

Block size is three decisions written as one number

Ratio rises with block size, because there is more window content and the entropy tables are shared by more data. It climbs steeply up to about 64–256 KB and then flattens. Below about 4 KB the table overhead dominates and the ratio falls toward 1.

Read amplification rises linearly with block size.

Reading one 100-byte row from a 1 MB block: 1,048,576 / 100 ≈ 10,000× amplification, and about 1 ms of decompression at 1 GB/s.
Reading the same row from a 4 KB block: 4,096 / 100 ≈ 40× amplification, and about 4 µs.

For a point-lookup workload this difference is the latency budget. This is the "granularity mismatch creates amplification" idea again. Here it is fully self-inflicted and has an explicit price, which makes it a good place to see that amplification can be a design parameter and not only a disease.

Parallelism needs independent blocks, and independence costs ratio, because each block starts its model with an empty window. This is a price paid on purpose, and it has a receipt: ZSTD's multi-threaded mode measurably loses ratio at frame boundaries, and the loss grows with the thread count.

So the best block size is where the marginal gain in ratio equals the marginal loss from amplification, and it depends only on the access pattern. Parquet pages at about 1 MB and RocksDB blocks at 4–16 KB are the two ends of the same equation with different inputs: scanning versus seeking. Neither is better. They are the right answers to different questions. Copying one system's block size into a workload with the other's access pattern is one of the most common and most expensive configuration mistakes in this area.

Checksums and corruption amplification

Compression turns a small, local error into the loss of a whole block.

A flipped bit in a match offset makes the decoder copy from the wrong place, and the error spreads forward through every later match.
A flipped bit in an entropy-coded stream shifts the bit boundary, and everything after it is noise.

The blast radius of an error is the block size. This is the same number as the read amplification and the window scope, for the third time.

Worse, compression makes corruption harder to notice. Compressed bytes look random (high entropy), so corrupted output does not look wrong to any simple check later in the pipeline. So every format keeps a checksum (a short value computed from the bytes, which changes if any byte changes) per block: Parquet has CRC32C per page, ZSTD has an optional XXH64 per frame, RocksDB has one per block. CRC32C has a dedicated CPU instruction since SSE4.2 and runs at 10–20 GB/s, which makes it free by any reasonable accounting.

"We disabled checksums for performance" is folklore from before that instruction existed. It deserves a firm correction, because what it switches off is the only detector of a failure that compression itself created.

Deduplication and content-defined chunking

Deduplication is a separate mechanism, not a variant of LZ. LZ finds redundancy inside a window. Deduplication finds it at scales no window can reach: across files, across snapshots, across a whole repository. It works in four steps:

Cut the data into chunks at boundaries chosen by a rolling hash of the content itself (a hash computed over a sliding window of bytes). This is content-defined chunking. Because the boundaries depend on the content, inserting a few bytes moves the boundary of only one chunk and not of every chunk after it.
Hash each chunk.
Store each distinct chunk once.
Keep a list of chunk hashes for each file.

This is how backup systems, container registries and version-controlled object stores reach ratios that no codec can reach. ZSTD's long-range mode is a partial, single-stream version of the same idea. Think of it as a fourth stage, after ZSTD: redundancy above the window scale.