Batch size is a compression parameter

Measure your machineNeeds Python with zstandard

Problem

Compression finds repetition, and a single small message has almost none: every field name appears once. A batch of events repeats the same field names, the same product ids and the same status values many times, so the compressor can encode them once. The batch size a pipeline chooses for throughput is therefore also its compression ratio. When messages must travel alone, a dictionary trained on typical messages gives the compressor the repetition in advance.

Generate 100,000 small JSON events of about 200 bytes with realistic repetition (same field names, a few dozen distinct product ids). Compress them with Zstd at different batch sizes and print the total output size. Then train a dictionary (zstd.train_dictionary(16384, events[:5000]), pass it to ZstdCompressor(dict_data=...)) and repeat for batch = 1.

The question. How does the compression ratio of the same events change from one event per frame to 10,000, and what does a trained dictionary do for single events?

Predict

With one event per frame, what compression ratio does zstd reach?

Nothing (under 1.1×)1.1 to 1.5×1.5 to 4×4× or more

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

batch-size-is-a-compression-parameter.py44 linesDownload

View the code

batch-size-is-a-compression-parameter.py

# /// script
# requires-python = ">=3.10"
# dependencies = ["zstandard"]
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Batch size is a compression parameter.

100,000 small JSON events of about 200 bytes with realistic repetition,
compressed with Zstd in batches of 1 to 10,000 events; then single events
compressed with a trained dictionary.
"""
import argparse, json, random
import zstandard as zstd

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--events", type=int, default=100_000, help="Events.")
p.add_argument("--level", type=int, default=3, help="Zstd level.")
p.add_argument("--sizes", type=lambda s: [int(v) for v in s.split(",")], default=[1, 10, 100, 1000, 10000], help="Batch sizes.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
random.seed(1)
events = [json.dumps({"order_id": 1_000_000 + i, "seller_id": random.randint(1, 40),
                      "sku": f"SKU-{random.randint(1, 60)}", "qty": random.randint(1, 5),
                      "status": "OrderPlaced", "country": random.choice(["IN", "UK", "US"])}).encode()
          for i in range(args.events)]
raw = sum(len(e) for e in events)
c = zstd.ZstdCompressor(level=args.level)
rows = []
for batch in args.sizes:
    out = sum(len(c.compress(b"\n".join(events[i:i + batch]))) for i in range(0, len(events), batch))
    rows.append({"batch": batch, "ratio": raw / out, "mode": "plain"}); print(f"batch={batch:6d}  ratio={raw / out:5.2f}x")
d = zstd.train_dictionary(16384, events[:5000])
cd = zstd.ZstdCompressor(level=args.level, dict_data=d)
out = sum(len(cd.compress(e)) for e in events[5000:]) * len(events) / (len(events) - 5000)
rows.append({"batch": 1, "ratio": raw / out, "mode": "dictionary"})
print(f"batch=     1  with a 16 KB trained dictionary  ratio={raw / out:5.2f}x   (average event {raw / len(events):.0f} bytes)")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"ratios": [r for r in rows if r["mode"] == "plain"], "summary": [{
            "single": rows[0]["ratio"], "k1000": next(r["ratio"] for r in rows if r["batch"] == 1000 and r["mode"] == "plain"),
            "dict": rows[-1]["ratio"], "avg_bytes": raw / len(events)}]}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.

Terminal

uv run batch-size-is-a-compression-parameter.py
uv run batch-size-is-a-compression-parameter.py --help     # every knob

Knobs
Knob · default	What it changes
--events100,000	Events.
--level3	Zstd level.
--sizes1, 10, 100, 1000, 10,000	Batch sizes.

Our run

Compression ratio by batch size (Zstd level 3)Apple M2 · 8 cores · 8 GB

Data table

Compression ratio by batch size (Zstd level 3)
Events per compressed frame	Compression ratio
1	1.03×
10	5.05×
100	11.2×
1K	12.2×
10K	12.2×

macOS 26.6.2 · Python 3.13.3 · zstandard 0.25.0 · run 2026-10-20

Output · uv run batch-size-is-a-compression-parameter.py

batch=     1  ratio= 1.03x
batch=    10  ratio= 5.05x
batch=   100  ratio=11.19x
batch=  1000  ratio=12.20x
batch= 10000  ratio=12.16x
batch=     1  with a 16 KB trained dictionary  ratio= 3.06x   (average event 107 bytes)

What we found

One event per frame compressed 1.03×: essentially nothing. Batches of 1,000 reached 12.2×, and larger batches added almost nothing more, as expected. A 16 KB dictionary trained on 5,000 events lifted single events to 3.06×. The events averaged 107 bytes, about half the 200 the lab suggests, which makes single events even harder to compress.

Read it

One decision at the top has consequences four layers down: the partition count caps parallelism, which sets batch sizes, which sets the compression ratio and the file sizes downstream. A pipeline that sends records one at a time pays for it twice, in fixed costs per call and in bytes on the wire. Where single messages are unavoidable, a shared dictionary recovers much of the loss, at the price of managing dictionary versions on both ends.