Fan-out arithmetic and rebalancing time

Simulate a modelNeeds Python with NumPy

Problem

When one request fans out to N backend calls and waits for all of them, it is as slow as its slowest call. If each call is slow with probability p (say, beyond its own 99th percentile, p = 0.01), the chance that at least one of N is slow is 1 − (1 − p)^N, which grows quickly with N. The second formula is simpler but just as often ignored: moving data between nodes (a rebalance) takes the data moved divided by the throttle, and the source copy stays on disk until the move completes.

Check two formulas from The arithmetic worth carrying and State has mass with real numbers.

Compute P(at least one slow of N) = 1 - (1-p)^N for p = 0.01 and N = 1, 10, 40, 100. Then simulate it: draw N latencies per request from a distribution whose 99th percentile is 50 ms, take the maximum, and print the p50 and p99 of the maximum over 100,000 requests.
Compute rebalancing time = data moved / throttle. For 10 TB at 100 MB/s: 10,000,000 MB / 100 = 100,000 s, about 27.8 hours. Change the throttle to 50 and 200 MB/s and note how the source disk footprint stays high for the whole window.

The question. For N = 40 calls per request, how many requests contain at least one slow call, and where does the request's median land against a single call's p99?

Predict

With 40 calls per request, each slow 1% of the time, what share of requests contain at least one slow call?

Under 5%5 to 20%20 to 50%Over half

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

fan-out-arithmetic-and-rebalancing-time.py47 linesDownload

View the code

fan-out-arithmetic-and-rebalancing-time.py

# /// script
# requires-python = ">=3.10"
# dependencies = ["numpy"]
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Fan-out arithmetic and rebalancing time.

P(at least one slow call of N) = 1 - (1 - p)^N, checked by simulation: N
calls per request, the request waits for the slowest. Then the time to move
data during a rebalance at a throttled rate.
"""
import argparse, json
import numpy as np

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--fanouts", type=lambda s: [int(v) for v in s.split(",")], default=[1, 10, 40, 100], help="Calls per request (N).")
p.add_argument("--requests", type=int, default=100_000, help="Simulated requests per N.")
p.add_argument("--p99-ms", type=float, default=50, help="99th percentile of one call's latency.")
p.add_argument("--data-tb", type=float, default=10, help="Data moved by the rebalance, in TB.")
p.add_argument("--throttles", type=lambda s: [float(v) for v in s.split(",")], default=[50, 100, 200], help="Rebalance throttles in MB/s.")
p.add_argument("--seed", type=int, default=0, help="Random seed.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
rng = np.random.default_rng(args.seed)

# one call: log-normal with median 10 ms, scaled so its p99 is p99_ms
sigma = np.log(args.p99_ms / 10) / 2.326
rows = []
for n in args.fanouts:
    calls = 10 * np.exp(sigma * rng.standard_normal((args.requests, n)))
    worst = calls.max(axis=1)
    rows.append({"N": n, "formula_pct": (1 - 0.99 ** n) * 100, "measured_pct": float(np.mean(worst > args.p99_ms) * 100),
                 "p50_ms": float(np.percentile(worst, 50)), "p99_ms": float(np.percentile(worst, 99))})
    r = rows[-1]
    print(f"N = {n:>3}: P(at least one call over p99) formula {r['formula_pct']:5.1f}%  measured {r['measured_pct']:5.1f}%   "
          f"request p50 {r['p50_ms']:5.1f} ms  p99 {r['p99_ms']:6.1f} ms")
reb = []
for mbps in args.throttles:
    s = args.data_tb * 1e6 / mbps
    reb.append({"throttle_mbps": mbps, "seconds": s, "hours": s / 3600})
    print(f"rebalance {args.data_tb:g} TB at {mbps:g} MB/s: {s:,.0f} s = {s / 3600:.1f} hours with the source copy still on disk")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"fanout": rows, "rebalance": reb}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.

Terminal

uv run fan-out-arithmetic-and-rebalancing-time.py
uv run fan-out-arithmetic-and-rebalancing-time.py --help     # every knob

Knobs
Knob · default	What it changes
--fanouts1, 10, 40, 100	Calls per request (N).
--requests100,000	Simulated requests per N.
--p99-ms50	99th percentile of one call's latency.
--data-tb10	Data moved by the rebalance, in TB.
--throttles50, 100, 200	Rebalance throttles in MB/s.
--seed0	Random seed.

Our run

Requests that contain at least one slow callApple M2 · 8 cores · 8 GB

Simulated1 − (1 − p)^N

Data table

Requests that contain at least one slow call
Calls per request (N)	Simulated	1 − (1 − p)^N
1	1.02%	1%
10	9.71%	9.56%
40	33.3%	33.1%
100	63.4%	63.4%

macOS 26.6.2 · Python 3.13.3 · numpy 2.5.4 · run 2026-10-20

The request waits for its slowest call

The request waits for its slowest call
N	At least one slow call	Request p50 (ms)	Request p99 (ms)
1	1.02%	9.97	50.2
10	9.71%	28.2	85
40	33.3%	43.3	111
100	63.4%	54.8	131

Rebalancing 10 TB

Rebalancing 10 TB
Throttle (MB/s)	Hours
50	55.6
100	27.8
200	13.9

Output · uv run fan-out-arithmetic-and-rebalancing-time.py

N =   1: P(at least one call over p99) formula   1.0%  measured   1.0%   request p50  10.0 ms  p99   50.2 ms
N =  10: P(at least one call over p99) formula   9.6%  measured   9.7%   request p50  28.2 ms  p99   85.0 ms
N =  40: P(at least one call over p99) formula  33.1%  measured  33.3%   request p50  43.3 ms  p99  111.0 ms
N = 100: P(at least one call over p99) formula  63.4%  measured  63.4%   request p50  54.8 ms  p99  131.2 ms
rebalance 10 TB at 50 MB/s: 200,000 s = 55.6 hours with the source copy still on disk
rebalance 10 TB at 100 MB/s: 100,000 s = 27.8 hours with the source copy still on disk
rebalance 10 TB at 200 MB/s: 50,000 s = 13.9 hours with the source copy still on disk

What we found

The simulation matched the formula at every N: at N = 40, 33.3% of requests contained at least one call slower than the single-call p99 (formula 33.1%). The request's median was 43.3 ms, close to the 50 ms single-call p99, as expected, and its own p99 was 111 ms. At N = 100 almost two requests in three contained a slow call (63.4%).

The rebalance arithmetic is exact: 10 TB at 100 MB/s takes 27.8 hours; halving the throttle doubles it to 55.6, and for that whole window the data exists twice.

Read it

A fan-out turns a per-call tail into a per-request median. The fix is structural: wait for fewer of the calls (a quorum waits for the median response, not the maximum), hedge the slow ones, or fan out less.

The rebalance throttle is a trade between how hard the move hits live traffic and how long the cluster runs with two copies of the moving data. A gentle throttle stretches the window in which disks are fuller and in which a second failure finds the data in transit.