Compression on the wire
When does compressing pay, and how is compression coupled to batching, partitioning and encryption?
Compression trades space for time: spend CPU to move fewer bytes. The basic trade is straightforward. The interactions with other mechanisms are not.
The decision is a rate comparison, not a ratio comparison
The question is never "which algorithm compresses best?" The question is:
So the answer depends on where your bottleneck is.
- Cross-region, internet, or metered egress (you pay per gigabyte leaving the cloud): the link is the bottleneck, so compress aggressively. Even slow algorithms win. Egress cost is real money, and compression is one of the highest-return cost levers in a data platform.
- Inside a datacenter at 25–100 Gbps: the bottleneck is often the CPU. A slow compression algorithm (a codec) becomes the bottleneck and you have made things worse. Use fast codecs only, or none.
- Loopback or same host: compression is pure loss.
(In practice the three steps overlap as a pipeline, so the real total is closer to the slowest step. The conclusion does not change: on a fast link the codec is the slow part.) On a metered link the winner is even clearer, because the bill falls to one third regardless of speed.
Codecs in practice
LZ4 and Zstd exist because they changed the answer: LZ4 compresses at multiple GB/s, nearly free compared with any network, and Zstd at levels 1–3 is the modern default for the wire (10 and above for storage). Gzip is for compatibility only.
The interactions: this is the part that matters
Compression and batching multiply. Compression finds redundancy across records. A batch of 1,000 similar records compresses far better than 1,000 records compressed one by one, often several times better. So your batch size is a compression parameter, and small batches silently destroy your compression ratio.
Example (illustrative). Lena's events are about 200 bytes each and look alike (same field names, similar values).
| How compressed | Input | Typical output | Ratio |
|---|---|---|---|
| Each event alone | 200 B | ~170 B (no history to refer to) | ~1.2× |
| A batch of 1,000 events | 200,000 B | ~30,000 B | ~6.7× |
For 1 million events per hour the alone-case sends about 170 MB, the batch case about 30 MB. That is the gap between a high bill and a low bill, from a configuration change.
The consequence people hit: someone lowers the producer's linger.ms (how long the producer waits to fill a batch) to improve latency. Batches shrink. The compression ratio collapses. Network and storage costs jump. Nobody connects the two changes, because the coupling between batching and compression is invisible in the configuration.
Compression and partitioning interact. More partitions mean smaller batches per partition, which means worse compression. Over-partitioning has a compression cost that nobody counts when choosing the number of partitions N (Routing and distribution).
Compression and columnar layout interact strongly. A column is homogeneous, so dictionary, run-length and delta encoding raise the ratio from about 2–3× for row layouts to 10× or more, and sorting before writing creates the runs that compress.
Compression and encryption interact dangerously. Compress-then-encrypt is the only order that saves bytes, but the encrypted size still reveals the compressed size, so never compress attacker-influenced data together with secrets. Why that leaks the secret, and the escapes, is security's subject.
Where to compress
| Level | Idea | Ratio | Notes |
|---|---|---|---|
| Per record | each record alone | worst | simple; allows independent access; almost always the wrong choice at volume |
| Per batch or block | one compressed unit holds many records | good | the standard: Kafka compresses the record batch; Parquet compresses each column chunk of each row group; the block is the unit of independent access |
| Stream level, shared window | the whole connection is one compressed stream | best | a lost or reordered chunk breaks everything after it, so the transport must be reliable and ordered |
| Dictionary-trained | pre-train a dictionary on a sample of your data and give it to both ends | dramatic on small messages | Zstd supports dictionaries |
Dictionary training deserves a note. Per-message compression normally fails on small messages, because there is no history to refer to. A shared dictionary is the history. It is underused for small-record streams. If your records are a few hundred bytes and you compress them one by one, this is the fix.
The end-to-end property
Kafka's design shows a pattern worth naming. The producer compresses the batch. The broker stores and forwards it compressed, without decompressing it (as long as the codecs are compatible). The consumer decompresses. That is one compression and one decompression in total, and the broker's CPU is not touched.
Break this and you pay. If the broker must recompress (the codec does not match, or it converts the format for an old client, or it rewrites timestamps), it adds a full decompress-and-recompress cycle for every message. This is a well-known way to saturate broker CPU without warning. The general lesson: compression should be end-to-end. Every intermediary that touches it costs a full round of CPU. The same is true of any proxy that decompresses to inspect a message and then compresses again.