Labs

Find your own break-even point D/B

Compress a 300 MB JSON log at zstd levels 1 to 19 and check, against a local disk and an object store, whether decoding beats reading.

How big is JSON really, after compression?

Encode 1,000 parcel-scan records as JSON and as Avro, raw and after zstd, with random and with sequential IDs.

Make `memcmp` sort your numbers correctly

Sort 10,000 integers and floats by their byte encodings, with and without the sign-bit flip and the float rule, and test composite-key escaping.

Varint and zigzag: where the bet wins and loses

Bytes per value under varint and zigzag + varint for small counts, IDs, random 64-bit values and small signed numbers.

Row order versus codec

Write 2 million rows to Parquet shuffled and sorted under NONE, SNAPPY and ZSTD; compare sizes and the depot column's row-group statistics.

One outlier sets the width

Bit-pack a block of 128 values, add one outlier, then patch it out with PFOR.

Encoding by hand: dictionary, RLE and delta

A 10-million-value country column and a sorted timestamp column: dictionary, RLE before and after sorting, delta, and gzip, computed by hand.

Are your floats decimals in disguise?

A million latitude-like doubles with six decimals: zlib on the raw bytes against scaling to integers and frame-of-reference.

Block size, ratio and point-lookup cost

Compress 200,000 JSON rows in blocks from 1 KB to 1 MB; compare ratio and the cost of decoding one block for a point lookup.

Does a trained dictionary rescue small messages?

Compress 100-byte JSON messages one at a time with and without a trained zstd dictionary, then decode with the wrong one.