Mean, percentile averaging and fan-out in 20 lines (30 minutes)
Simulate a modelNeeds Python with NumPy
Problem
Latency distributions are skewed, and the usual summaries mislead in specific ways. The mean is pulled by the slow tail to a value no request experienced. A percentile cannot be averaged across shards. And a request that fans out to many calls and waits for all of them takes the slowest one: with 100 calls, the chance that at least one lands in the slowest 1% is 1 − 0.99¹⁰⁰ ≈ 63%, so the service's typical request lives at the components' p99.
Generate 1,000,000 latencies: 99% from a normal distribution around 5 ms and 1% around 700 ms. Print the mean, p50, p99 and p99.9. Then split the data into 10 unequal shards, compute each shard's p99, average them, and compare with the global p99. Finally simulate a fan-out: for each request, take the maximum of 100 independent draws.
The question. What do the mean and percentiles say, how wrong is the averaged p99, and what share of fan-out requests is slower than one call's p99?
Predict
A mean near 12 ms that no request experienced, an average-of-shard-p99 that differs from the true p99, and a fan-out fraction near 63% (1 − 0.99^100 ≈ 0.634).
Run it
1Install uv once. It fetches Python and the lab's packages for you.
Terminal curl -LsSf https://astral.sh/uv/install.sh | sh2Get the script. It is the whole lab: read it, then download it.
View the codeHide the code
mean-percentile-averaging-and-fan-out.py # /// script # requires-python = ">=3.10" # dependencies = ["numpy"] # /// # Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT # https://thedataengineering.com/lab-code/LICENSE.txt """Mean, percentile averaging and fan-out. One million latencies: 99% around 5 ms, 1% around 700 ms. Prints the mean and percentiles, the average of ten unequal shards' p99 against the global p99, and the share of fan-out requests (the maximum of 100 draws) slower than the single-call p99. """ import argparse, json import numpy as np p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) p.add_argument("--n", type=int, default=1_000_000, help="Latencies to draw.") p.add_argument("--fanout", type=int, default=100, help="Calls per fanned-out request.") p.add_argument("--slow-share", type=float, default=0.01, help="Share of slow calls.") p.add_argument("--seed", type=int, default=1, help="Random seed.") p.add_argument("--json", help="Also write the results to this JSON file.") args = p.parse_args() rng = np.random.default_rng(args.seed) def draw(n): slow = rng.random(n) < args.slow_share return np.where(slow, rng.normal(700, 80, n), rng.normal(5, 1, n)).clip(0.5) x = draw(args.n) p50, p99, p999 = np.percentile(x, [50, 99, 99.9]) stats = [{"statistic": k, "ms": float(v)} for k, v in (("mean", x.mean()), ("p50", p50), ("p99", p99), ("p99.9", p999))] # ten unequal shards, and the slow calls are not spread evenly across them (they lean to the later shards) cuts = np.sort(rng.choice(np.arange(1, args.n), 9, replace=False)) skewed = np.split(x[np.argsort(rng.random(args.n) + (x > 100) * rng.random(args.n))], cuts) shard_rows = [{"shard": i + 1, "rows": len(s), "p99_ms": float(np.percentile(s, 99))} for i, s in enumerate(skewed)] avg_p99 = float(np.mean([r["p99_ms"] for r in shard_rows])) fan = np.concatenate([draw(10_000 * args.fanout).reshape(10_000, args.fanout).max(axis=1) for _ in range(10)]) share = float((fan > p99).mean()); theory = 1 - (1 - args.slow_share) ** args.fanout summary = {"mean": float(x.mean()), "p50": float(p50), "p99": float(p99), "p999": float(p999), "avg_shard_p99": avg_p99, "fan_share": share, "fan_theory": theory, "fan_p50": float(np.percentile(fan, 50))} print(f"mean {x.mean():.1f} ms p50 {p50:.1f} p99 {p99:.1f} p99.9 {p999:.1f}") print(f"average of ten shard p99s {avg_p99:.1f} ms against the global p99 {p99:.1f} ms") print(f"fan-out of {args.fanout}: {share:.3f} of requests slower than the single-call p99 (1 - 0.99^{args.fanout} = {theory:.3f}); median request {summary['fan_p50']:.0f} ms") if args.json: with open(args.json, "w") as f: json.dump({"summary": summary, "tables": {"stats": stats, "shards": shard_rows}}, f)3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal uv run mean-percentile-averaging-and-fan-out.py uv run mean-percentile-averaging-and-fan-out.py --help # every knob
| Knob · default | What it changes |
|---|---|
| --n1,000,000 | Latencies to draw. |
| --fanout100 | Calls per fanned-out request. |
| --slow-share0.01 | Share of slow calls. |
| --seed1 | Random seed. |
Our run
One million latencies
| Statistic | ms |
|---|---|
| mean | 12 |
| p50 | 5.01 |
| p99 | 484 |
| p99.9 | 803 |
mean 12.0 ms p50 5.0 p99 483.9 p99.9 803.0
average of ten shard p99s 84.5 ms against the global p99 483.9 ms
fan-out of 100: 0.633 of requests slower than the single-call p99 (1 - 0.99^100 = 0.634); median request 661 msWhat we found
The mean was 12 ms, between a median of 5.01 ms and a p99 of 484 ms: no request took that long. The average of the ten shard p99s was 84.5 ms against a true p99 of 484 ms: the shards split the slow calls unevenly, and averaging their p99s does not reconstruct the tail. In the fan-out, 0.633 of requests were slower than a single call's p99 (theory 0.634), and the median request took 661 ms.
Read it
Report latency as percentiles from the full distribution, and compute them from merged data, never from averaged summaries. For fan-out systems the component that matters is the tail: a call's p99 becomes the median of a request that waits for a hundred of them.