Does a trained dictionary rescue small messages?

Measure your machineNeeds Python with zstandard

Problem

A compressor learns the data as it goes, so a small message ends before it has learned anything: compressed alone, a 100-byte message barely shrinks, because the description of its patterns is larger than the data. A trained dictionary gives the compressor that knowledge in advance, built from samples of similar messages and shared by both sides. The cost moves from size to dependency: every reader must hold the exact dictionary each message was written with, forever.

See the "description larger than the data" problem, and its cost. Train a 16 KB dictionary on 10,000 messages, compress the other 10,000 one at a time with and without it, then try to decompress one message with a different dictionary or none.

The question. How much does a trained dictionary improve the ratio on small messages, and what happens without it?

Predict

With a trained 16 KB dictionary, how much does zstd compress 100-byte messages?

Under 1.5×1.5 to 2×2 to 5×5× or more

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

does-a-trained-dictionary-rescue-small.py48 linesDownload

View the code

does-a-trained-dictionary-rescue-small.py

# /// script
# requires-python = ">=3.10"
# dependencies = ["zstandard"]
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Does a trained dictionary rescue small messages?

Small JSON messages compressed one at a time with Zstd, without and with a
dictionary trained on 10,000 samples; then one message decompressed with the
wrong dictionary, and with none.
"""
import argparse, json, random
import zstandard as zstd

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--messages", type=int, default=20_000, help="Messages (half train the dictionary).")
p.add_argument("--dict-kb", type=int, default=16, help="Dictionary size, KB.")
p.add_argument("--seed", type=int, default=0, help="Random seed.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
random.seed(args.seed)
def msg(i):
    return json.dumps({"parcel_id": f"P{random.randint(0, 10**7)}", "scanner_id": f"S{random.randint(1, 900)}",
                       "status": random.choice(["SCANNED", "LOADED", "DELIVERED", "DELAYED"]),
                       "depot": random.choice(["RTM", "LYN", "HAM", "WAW"]), "weight_grams": random.randint(100, 30000)}).encode()
samples = [msg(i) for i in range(args.messages)]
half = args.messages // 2
d = zstd.train_dictionary(args.dict_kb * 1024, samples[:half])
plain, withd = zstd.ZstdCompressor(level=3), zstd.ZstdCompressor(level=3, dict_data=d)
test = samples[half:]; raw = sum(len(m) for m in test)
r_plain = raw / sum(len(plain.compress(m)) for m in test); r_dict = raw / sum(len(withd.compress(m)) for m in test)
print(f"no dictionary : {r_plain:.2f}x   (average message {raw / len(test):.0f} bytes)")
print(f"dictionary    : {r_dict:.2f}x")
blob = withd.compress(test[0])
other = zstd.train_dictionary(args.dict_kb * 1024, [msg(i) for i in range(half)])
outcomes = {}
for label, dec in (("the right dictionary", zstd.ZstdDecompressor(dict_data=d)), ("a different dictionary", zstd.ZstdDecompressor(dict_data=other)),
                   ("no dictionary", zstd.ZstdDecompressor())):
    try: outcomes[label] = "ok" if dec.decompress(blob) == test[0] else "wrong bytes"
    except zstd.ZstdError as e: outcomes[label] = f"error: {str(e)[:60]}"
    print(f"decompress with {label}: {outcomes[label]}")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"summary": [{"plain": r_plain, "dict": r_dict, "avg_bytes": raw / len(test)}],
                   "decode": [{"decoder": k, "outcome": v} for k, v in outcomes.items()]}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.

Terminal

uv run does-a-trained-dictionary-rescue-small.py
uv run does-a-trained-dictionary-rescue-small.py --help     # every knob

Knobs
Knob · default	What it changes
--messages20,000	Messages (half train the dictionary).
--dict-kb16	Dictionary size, KB.
--seed0	Random seed.

Our run

Decompressing one message

Decompressing one message
Decoder	Outcome
the right dictionary	ok
a different dictionary	error: decompression error: Dictionary mismatch
no dictionary	error: decompression error: Dictionary mismatch

Reference runApple M2 · 8 cores · 8 GB

Output · uv run does-a-trained-dictionary-rescue-small.py

no dictionary : 1.02x   (average message 107 bytes)
dictionary    : 2.91x
decompress with the right dictionary: ok
decompress with a different dictionary: error: decompression error: Dictionary mismatch
decompress with no dictionary: error: decompression error: Dictionary mismatch

macOS 26.6.2 · Python 3.13.3 · zstandard 0.25.0 · run 2026-10-20

What we found

Messages of about 107 bytes compressed 1.02× alone and 2.91× with the dictionary, inside both expected ranges. With a different dictionary the decoder reported error: decompression error: Dictionary mismatch; with none, error: decompression error: Dictionary mismatch.

Read it

A dictionary turns many small messages into one large compression context, shared out of band. That is the right move when the messages are small and similar and the dictionary can be versioned like a schema: each message must say which dictionary it needs, and every dictionary ever used must stay readable for as long as its messages exist.