Row order versus codec

Measure your machineNeeds Python with PyArrow, NumPy

Problem

A columnar file compresses in two stages: encodings that exploit structure (dictionary for repeated values, run-length for runs, delta for slowly changing numbers), then a general-purpose codec over the result. The encodings only find structure that the row order exposes: sorting a table makes equal values adjacent, which turns dictionary indexes into long runs and timestamps into small deltas. Order also decides what the row-group statistics can skip.

Show that sort order changes file size more than the codec does. The script writes 2 million rows (200 depots, 10 statuses, an increasing event time) shuffled and sorted by (depot, event_time), with each codec, and then reads the min/max of depot per row group in both files.

The question. Does sorting the rows shrink the file more than switching codecs, and what does it do to the row-group statistics?

Predict

With ZSTD, how much smaller does sorting the rows make the Parquet file?

Under 1.5×1.5 to 3×3 to 10×10× or more

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

row-order-versus-codec.py44 linesDownload

View the code

row-order-versus-codec.py

# /// script
# requires-python = ">=3.10"
# dependencies = ["numpy", "pyarrow"]
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Row order against the codec.

The same 2 million rows (depot, status, event_time) written to Parquet
shuffled and sorted by depot then time, with no codec, Snappy and Zstd; then
the depot column's min/max per row group in each file.
"""
import argparse, json, os, tempfile
import numpy as np, pyarrow as pa, pyarrow.parquet as pq

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--rows", type=int, default=2_000_000, help="Rows.")
p.add_argument("--row-group", type=int, default=128_000, help="Rows per row group.")
p.add_argument("--seed", type=int, default=1, help="Random seed.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
rng = np.random.default_rng(args.seed); N = args.rows
tbl = pa.table({"depot": rng.integers(0, 200, N), "status": rng.integers(0, 10, N), "event_time": np.cumsum(rng.integers(1, 5_000, N))})
d = tempfile.mkdtemp()
def size(table, name, codec):
    path = os.path.join(d, name); pq.write_table(table, path, compression=codec, use_dictionary=True, row_group_size=args.row_group)
    return path, os.path.getsize(path) / 1e6
shuffled = tbl.take(rng.permutation(N)); sorted_ = tbl.sort_by([("depot", "ascending"), ("event_time", "ascending")])
rows = []
for codec in ("NONE", "SNAPPY", "ZSTD"):
    _, a = size(shuffled, f"a-{codec}.parquet", codec); _, b = size(sorted_, f"b-{codec}.parquet", codec)
    rows.append({"codec": codec, "shuffled_mb": a, "sorted_mb": b, "ratio": a / b})
    print(f"{codec:<7} shuffled {a:6.1f} MB   sorted {b:6.1f} MB   ({a / b:.1f}x)")
widths = {}
for label, name in (("shuffled", "a-ZSTD.parquet"), ("sorted", "b-ZSTD.parquet")):
    md = pq.read_metadata(os.path.join(d, name))
    w = [md.row_group(i).column(0).statistics for i in range(md.num_row_groups)]
    widths[label] = sum(s.max - s.min for s in w) / len(w)
    print(f"{label:<9} depot min/max per row group: average width {widths[label]:.0f} of 0-199  (first: {w[0].min}-{w[0].max})")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"codecs": rows, "summary": [{"shuffled_width": widths["shuffled"], "sorted_width": widths["sorted"]}]}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run row-order-versus-codec.py
uv run row-order-versus-codec.py --help     # every knob
```

Knobs
Knob · default	What it changes
--rows2,000,000	Rows.
--row-group128,000	Rows per row group.
--seed1	Random seed.

Our run

File size by order and codec (MB)

File size by order and codec (MB)
Codec	Shuffled	Sorted	Shuffled ÷ sorted
NONE	23.2	21.1	1.1×
SNAPPY	20	17.1	1.17×
ZSTD	16.2	12	1.35×

Reference runApple M2 · 8 cores · 8 GB

Output · uv run row-order-versus-codec.py

NONE    shuffled   23.2 MB   sorted   21.1 MB   (1.1x)
SNAPPY  shuffled   20.0 MB   sorted   17.1 MB   (1.2x)
ZSTD    shuffled   16.2 MB   sorted   12.0 MB   (1.3x)
shuffled  depot min/max per row group: average width 199 of 0-199  (first: 0-199)
sorted    depot min/max per row group: average width 12 of 0-199  (first: 0-12)

macOS 26.6.2 · Python 3.13.3 · numpy 2.5.4 · pyarrow 26.0.0 · run 2026-10-20

What we found

Sorting shrank the file, but not by several times: 1.1× with no codec, 1.35× with ZSTD. The codec mattered more: on the sorted file ZSTD took it from 21.1 MB to 12 MB. The reason is the event time: its increments are random up to 5,000 µs, so it stays a high-entropy column after sorting, and it is most of the file. The statistics behaved as expected: in the sorted file a row group's depot range averaged 12.4 values wide, in the shuffled file 199, the whole domain.

Read it

Order helps the columns it makes repetitive, and this table's largest column is not one of them. The expectation holds for tables dominated by low-cardinality or slowly changing columns, which is common, and fails when a high-entropy column dominates the bytes. The pruning effect does not depend on that: sorting by a column always narrows its statistics, which is why sort order is chosen for the filters as much as for the size.