Measure your skew distribution and price δ

Measure your machineNeeds Python with NumPy

Problem

Event-time processing waits a delay δ after an event time before treating it as complete; records that arrive later than that are late. Choosing δ means choosing a point on the skew distribution (arrival time minus event time): the share of records beyond it is the late rate, and δ itself is added to every result's latency. Skew is usually heavy-tailed, so each additional nine of completeness costs far more waiting than the one before.

Generate event times and arrival times, compute skew and its p50, p99, p99.9 and p99.99. Then compute, for each quantile used as δ, the fraction of late records and the extra waiting time. The skew is drawn from a Pareto distribution (mostly small, sometimes very large).

The question. How far apart are the skew quantiles, and what does each extra nine of completeness cost in waiting?

Predict

Going from 99.9% to 99.99% complete, how much longer must a window wait?

SecondsMinutesAbout an hourHours

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

measure-your-skew-distribution-and-price.py37 linesDownload

View the code

measure-your-skew-distribution-and-price.py

# /// script
# requires-python = ">=3.10"
# dependencies = ["numpy"]
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Measure your skew distribution, and price each nine.

One million arrival skews (arrival time minus event time) from a heavy-tailed
distribution. The p50, p99, p99.9 and p99.99 of the skew; for each, used as
the watermark delay δ, the share of records that arrive late and the extra
time every window waits.
"""
import argparse, json
import numpy as np

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--n", type=int, default=1_000_000, help="Records.")
p.add_argument("--shape", type=float, default=1.2, help="Pareto shape of the skew.")
p.add_argument("--scale", type=float, default=2.0, help="Skew scale (s).")
p.add_argument("--seed", type=int, default=0, help="Random seed.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()

skew = np.random.default_rng(args.seed).pareto(args.shape, args.n) * args.scale    # heavy-tailed: mostly small, sometimes very large
rows = []
for q in (50, 99, 99.9, 99.99):
    d = float(np.percentile(skew, q))
    rows.append({"quantile": f"p{q:g}", "delta_s": d, "late_pct": float((skew > d).mean() * 100)})
for i, r in enumerate(rows): r["extra_wait_s"] = r["delta_s"] - (rows[i - 1]["delta_s"] if i else 0)
summary = {"p9999_over_p50": rows[-1]["delta_s"] / rows[0]["delta_s"]}
for r in rows: print(f"δ = {r['quantile']:<6} = {r['delta_s']:10.1f} s   late records {r['late_pct']:6.3f}%   extra wait over the previous nine {r['extra_wait_s']:10.1f} s")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"summary": summary, "tables": {"quantiles": rows}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.

Terminal

uv run measure-your-skew-distribution-and-price.py
uv run measure-your-skew-distribution-and-price.py --help     # every knob

Knobs
Knob · default	What it changes
--n1,000,000	Records.
--shape1.2	Pareto shape of the skew.
--scale2	Skew scale (s).
--seed0	Random seed.

Our run

Each quantile as the watermark delay

Each quantile as the watermark delay
δ at	δ	Late records	Extra wait over the previous
p50	1.56 s	50%	1.56 s
p99	91.5 s	1%	89.9 s
p99.9	655 s	0.1%	563 s
p99.99	4,100 s	0.01%	3,440 s

Reference runApple M2 · 8 cores · 8 GB

Output · uv run measure-your-skew-distribution-and-price.py

δ = p50    =        1.6 s   late records 50.000%   extra wait over the previous nine        1.6 s
δ = p99    =       91.5 s   late records  1.000%   extra wait over the previous nine       89.9 s
δ = p99.9  =      655.0 s   late records  0.100%   extra wait over the previous nine      563.5 s
δ = p99.99 =     4095.8 s   late records  0.010%   extra wait over the previous nine     3440.8 s

macOS 26.6.2 · Python 3.13.3 · numpy 2.5.4 · run 2026-10-20

What we found

The median skew was 1.56 s; p99 was 91.5 s, p99.9 655 s and p99.99 4,100 s, 2,620× the median. Going from 99% to 99.9% complete cost 563 s of extra waiting on every window; going to 99.99% cost 3,440 s more.

Read it

There is no δ that makes results both prompt and complete when skew is heavy-tailed. Choose the error rate the consumers can accept, set δ from the measured quantile, and handle what arrives later with an explicit late-data path (allowed lateness, corrections) rather than a longer wait for everyone.