Measure key skew and the effect of salting

Simulate a modelNeeds Python

Problem

Partitioning by key keeps all of a key's records together, which gives per-key order and local state. It also hands you the key distribution: if one seller owns 40% of the events, the partition holding that seller gets 40% of the work, however many partitions there are. Partitioning by a finer key (the order id) spreads the load but gives up per-seller grouping. Salting the hot key (appending a small random number) splits only the hot seller across a few partitions, at the cost of a second step that adds the partial results back together.

Generate 1,000,000 synthetic order events for 500 sellers where one seller ("FreshBox") owns 40% of the records and the other 499 share the rest evenly. Assign each record to one of 12 partitions with hash(key) % 12, first with the key seller_id, then with order_id, then with seller_id plus a salt in 0..3 for FreshBox only. Print the records per partition and the ratio max/mean. Then compute the per-seller count with two-phase aggregation (count per salted key, then sum the four partials) and check that it equals the true count.

The question. How unbalanced are 12 partitions under each key, and does two-phase aggregation over the salted key give the right count?

Predict

After salting the hot seller four ways, how loaded is the busiest partition against the mean?

About even (under 1.5×)1.5 to 2.5×2.5 to 4×Unchanged (4× or more)

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

measure-key-skew-and-the-effect.py44 linesDownload

View the code

measure-key-skew-and-the-effect.py

# /// script
# requires-python = ">=3.10"
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Measure key skew and the effect of the partition key.

1,000,000 order events for 500 sellers, one of which owns 40% of them,
partitioned 12 ways by seller_id, by order_id, and by seller_id with a salt
for the hot seller; then two-phase aggregation of the salted counts.
"""
import argparse, json, random, zlib, collections

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--events", type=int, default=1_000_000, help="Order events.")
p.add_argument("--partitions", type=int, default=12, help="Partitions.")
p.add_argument("--hot-share", type=float, default=0.40, help="Share of events owned by the hot seller.")
p.add_argument("--salts", type=int, default=4, help="Salts for the hot seller.")
p.add_argument("--seed", type=int, default=0, help="Random seed.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
random.seed(args.seed)
N = args.partitions
def part(key): return zlib.crc32(key.encode()) % N

sellers = ["FreshBox" if random.random() < args.hot_share else f"s{random.randrange(499)}" for _ in range(args.events)]
keyings = {"seller_id": lambda s, i: s, "order_id": lambda s, i: f"order-{i}",
           "seller_id + salt for the hot seller": lambda s, i: f"{s}#{random.randrange(args.salts)}" if s == "FreshBox" else s}
rows, salted_counts = [], collections.Counter()
for name, fn in keyings.items():
    c = collections.Counter()
    for i, s in enumerate(sellers):
        k = fn(s, i); c[part(k)] += 1
        if name.startswith("seller_id +") and s == "FreshBox": salted_counts[k] += 1
    mean = args.events / N
    rows.append({"key": name, "max_over_mean": max(c.values()) / mean, "largest_share_pct": max(c.values()) / args.events * 100})
    print(f"{name:<38} max/mean = {rows[-1]['max_over_mean']:.2f}   largest partition {rows[-1]['largest_share_pct']:.1f}% of events")
true = sellers.count("FreshBox"); two_phase = sum(salted_counts.values())
print(f"FreshBox count: true {true:,}; two-phase (sum of {len(salted_counts)} salted partials) {two_phase:,}")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"keys": rows, "summary": [{"true": true, "two_phase": two_phase}]}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run measure-key-skew-and-the-effect.py
uv run measure-key-skew-and-the-effect.py --help     # every knob
```

Knobs
Knob · default	What it changes
--events1,000,000	Order events.
--partitions12	Partitions.
--hot-share0.4	Share of events owned by the hot seller.
--salts4	Salts for the hot seller.
--seed0	Random seed.

Our run

1,000,000 events over 12 partitions

1,000,000 events over 12 partitions
Partition key	Max / mean	Largest partition
seller_id	5.32×	44.4%
order_id	1×	8.37%
seller_id + salt for the hot seller	3.06×	25.5%

Reference runApple M2 · 8 cores · 8 GB

Output · uv run measure-key-skew-and-the-effect.py

seller_id                              max/mean = 5.32   largest partition 44.4% of events
order_id                               max/mean = 1.00   largest partition 8.4% of events
seller_id + salt for the hot seller    max/mean = 3.06   largest partition 25.5% of events
FreshBox count: true 400,197; two-phase (sum of 4 salted partials) 400,197

macOS 26.6.2 · Python 3.13.3 · run 2026-10-20

What we found

Keyed by seller, the largest partition held 44.4% of the events, 5.32× the mean, as expected. Keyed by order, the ratio was 1×. Salting FreshBox four ways cut it to 3.06×: better, but four salts still leave each salted part of a 40% seller at 10%. Two-phase aggregation gave 400K for FreshBox, exactly the true count of 400K.

Read it

Choosing key partitioning gives you local state and order, and hands you skew, which no amount of flow control fixes. Salt with enough parts to bring the hot key down to a fair share (here about 40% ÷ 8% = 5 or more), and only where the aggregation is associative, so the partials can be combined.