Do not average percentiles

Simulate a modelNeeds Python

Problem

A percentile is not additive: the p99 of a whole is not the average of its parts' p99s. The parts can differ in size (an average gives a shard of 100 values the same weight as one of 10,000) and in shape (a shard with a fat tail sets most of the pooled tail). The correct answer needs all the values, or a mergeable sketch: a summary that can be combined across shards and still answer quantile queries with a known error, such as KLL, t-digest or REQ.

Create 100 "shards" with 10,000 latency values each from different log-normal distributions. Compute p99 per shard, then (a) average the 100 p99 values, (b) compute the exact p99 of the pooled values, (c) merge 100 sketches and read p99. Try shard sizes that differ by 100 times.

The question. How far is the average of shard p99s from the pooled p99, and how close do merged sketches come?

Predict

With 100 equal-sized shards, how far off the pooled p99 is the average of the shard p99s?

Within 2%2 to 10% low10 to 50% lowOff by a factor of 2 or more

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

do-not-average-percentiles.py49 linesDownload

View the code

do-not-average-percentiles.py

# /// script
# requires-python = ">=3.10"
# dependencies = ["numpy", "datasketches"]
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Do not average percentiles.

100 shards of latencies from different log-normal distributions and sizes
that differ up to 100 times. The p99 of the whole fleet three ways: the
average of the shards' p99s, the exact p99 of the pooled values, and merged
sketches: KLL, t-digest, and REQ in high-rank-accuracy mode.
"""
import argparse, json
import numpy as np
from datasketches import kll_floats_sketch, req_floats_sketch, tdigest_double

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--shards", type=int, default=100, help="Shards.")
p.add_argument("--base-size", type=int, default=10_000, help="Values per shard when sizes are equal.")
p.add_argument("--seed", type=int, default=0, help="Random seed.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
rng = np.random.default_rng(args.seed)

rows = []
for label, sizes in (("equal sizes", [args.base_size] * args.shards),
                     ("sizes differ up to 100x", list(np.geomspace(1_000, 100_000, args.shards).astype(int)))):
    shards = [rng.lognormal(np.log(20) + rng.normal(0, 0.5), rng.uniform(0.3, 1.2), n) for n in sizes]
    avg = float(np.mean([np.percentile(s, 99) for s in shards]))
    exact = float(np.percentile(np.concatenate(shards), 99))
    est = {}
    for name, make in (("kll", lambda: kll_floats_sketch(200)), ("tdigest", lambda: tdigest_double(200)),
                       ("req", lambda: req_floats_sketch(30, True))):
        merged = make()
        for s in shards:
            k = make()
            for x in s: k.update(float(x))
            merged.merge(k)
        est[name] = merged.get_quantile(0.99)
    rows.append({"shards": label, "average_of_p99s": avg, "exact_p99": exact, **{f"{k}_p99": v for k, v in est.items()},
                 "avg_err_pct": (avg / exact - 1) * 100, **{f"{k}_err_pct": (v / exact - 1) * 100 for k, v in est.items()}})
    print(f"{label:<24} exact p99 {exact:7.1f} ms   average of p99s {avg:7.1f} ({avg / exact - 1:+.0%})   merged: "
          + "   ".join(f"{k} {v:7.1f} ({v / exact - 1:+.2%})" for k, v in est.items()))

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"methods": rows}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run do-not-average-percentiles.py
uv run do-not-average-percentiles.py --help     # every knob
```

Knobs
Knob · default	What it changes
--shards100	Shards.
--base-size10,000	Values per shard when sizes are equal.
--seed0	Random seed.

Our run

p99 of 100 shards, five ways (ms)

p99 of 100 shards, five ways (ms)
Shards	Exact	Average of p99s	KLL	t-digest	REQ
equal sizes	167	131	182	165	168
sizes differ up to 100x	181	156	179	181	181

Reference runApple M2 · 8 cores · 8 GB

Output · uv run do-not-average-percentiles.py

equal sizes              exact p99   167.4 ms   average of p99s   130.7 (-22%)   merged: kll   181.6 (+8.48%)   tdigest   164.5 (-1.72%)   req   167.5 (+0.05%)
sizes differ up to 100x  exact p99   180.8 ms   average of p99s   156.2 (-14%)   merged: kll   179.4 (-0.78%)   tdigest   181.0 (+0.11%)   req   180.5 (-0.15%)

macOS 26.6.2 · Python 3.13.3 · numpy 2.5.4 · datasketches 5.2.0 · run 2026-10-20

What we found

With equal shard sizes the average of p99s was -21.9% off the exact value; with sizes differing up to 100 times it was -13.6% off. That is wrong, but not "by a large factor": averaging under-weighted the shards with the fattest tails by tens of percent, not multiples. The sketches split by kind. REQ, which is built for tail accuracy, was within 0.054% and -0.15%; t-digest within -1.72% and 0.115%; KLL, whose error is uniform over ranks rather than tight at the tail, was 8.48% and -0.775% off.

Read it

The fraction-of-a-percent promise holds for sketches whose accuracy is concentrated where you query. KLL bounds the error in rank, ±ε·N everywhere, which near the tail translates into a large error in value. For p99 and beyond, use a relative-error sketch (REQ, or t-digest in practice).

Averaged percentiles show up on every dashboard that rolls up per-host latency. The number looks plausible, moves in the right direction, and is not the p99 of anything.