Recovery time arithmetic and replay storms

Simulate a modelNeeds Python

Problem

After an outage, a queue holds the work that arrived while it was down, and it can drain that backlog only with its spare capacity, the part not taken by new arrivals. An outage of T at utilisation ρ takes T · ρ / (1 − ρ) to drain: the busier the system, the longer the recovery, without bound as ρ approaches 1. While it drains, it sends work downstream at full speed, which can overload a stage sized for normal traffic: a replay storm. Dropping work that is too old to matter shortens the drain.

Simulate a queue with service rate C = 100 jobs/s and arrival rate λ = 80/s (ρ = 0.8). Stop service for 60 s, then restart it at full speed. Measure the time to drain the backlog. Repeat for ρ = 0.5 and ρ = 0.9. Then add a downstream stage with capacity 120 jobs/s that receives the catch-up at 100/s plus its own new load of 40/s, and see where it overloads. Finally add a rule that drops items older than 30 s and compare the recovery time.

The question. How long does the backlog take to drain at each utilisation, does the downstream stage survive the catch-up, and what does dropping stale items save?

Predict

During the catch-up at ρ = 0.8, how much load does the downstream stage get, against its capacity of 120 jobs a second?

Under 120: it copes120 to 160: overloadedOver 160

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

recovery-time-arithmetic-and-replay-storms.py58 linesDownload

View the code

recovery-time-arithmetic-and-replay-storms.py

# /// script
# requires-python = ">=3.10"
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Recovery-time arithmetic and replay storms.

A queue served at C = 100 jobs/s with arrivals at rho x C; service stops for
60 s, then restarts at full speed: time to drain, against T x rho / (1 - rho).
Then a downstream stage of 120 jobs/s receiving the catch-up plus its own
load; then dropping items older than 30 s.
"""
import argparse, json

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--capacity", type=float, default=100, help="C: jobs per second.")
p.add_argument("--outage", type=float, default=60, help="Seconds without service.")
p.add_argument("--rhos", type=lambda s: [float(v) for v in s.split(",")], default=[0.5, 0.8, 0.9], help="Utilizations.")
p.add_argument("--downstream", type=float, default=120, help="Downstream capacity, jobs per second.")
p.add_argument("--downstream-own", type=float, default=40, help="Downstream's own new load, jobs per second.")
p.add_argument("--max-age", type=float, default=30, help="Drop items older than this, seconds.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
C, T, DT = args.capacity, args.outage, 0.1

def drain(rho, max_age=None):
    from collections import deque
    q, t, lam, drained, down_peak = deque(), 0.0, rho * C, None, 0.0
    carry = 0.0
    while t < 5000:
        carry += lam * DT
        while carry >= 1: q.append(t); carry -= 1
        if max_age is not None:
            while q and t - q[0] > max_age: q.popleft()
        served = 0
        if t >= T:
            for _ in range(int(C * DT)):
                if q: q.popleft(); served += 1
        down_load = served / DT + args.downstream_own
        down_peak = max(down_peak, down_load)
        if t > T and not q and drained is None: drained = t - T
        if drained is not None: break
        t += DT
    return drained, down_peak

rows = []
for rho in args.rhos:
    d, peak = drain(rho)
    rows.append({"rho": rho, "drain_s": d, "formula_s": T * rho / (1 - rho), "downstream_peak": peak,
                 "downstream_overloaded": "yes" if peak > args.downstream else "no"})
    print(f"rho = {rho}: drained after {d:6.1f} s (formula {T * rho / (1 - rho):6.1f} s); downstream peak {peak:.0f}/s vs {args.downstream:.0f}/s capacity")
d, _ = drain(0.8, args.max_age)
print(f"rho = 0.8, dropping items older than {args.max_age:g} s: drained after {d:.1f} s")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"rhos": rows, "summary": [{"drop_drain_s": d, "max_age": args.max_age}]}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.

Terminal

uv run recovery-time-arithmetic-and-replay-storms.py
uv run recovery-time-arithmetic-and-replay-storms.py --help     # every knob

Knobs
Knob · default	What it changes
--capacity100	C: jobs per second.
--outage60	Seconds without service.
--rhos0.5, 0.8, 0.9	Utilizations.
--downstream120	Downstream capacity, jobs per second.
--downstream-own40	Downstream's own new load, jobs per second.
--max-age30	Drop items older than this, seconds.

Our run

Drain time after a 60 s outage

Drain time after a 60 s outage
ρ	Drain	Formula	Downstream peak (/s)	Downstream overloaded
0.5	59.9 s	60 s	140	yes
0.8	240 s	240 s	140	yes
0.9	540 s	540 s	140	yes

Reference runApple M2 · 8 cores · 8 GB

Output · uv run recovery-time-arithmetic-and-replay-storms.py

rho = 0.5: drained after   59.9 s (formula   60.0 s); downstream peak 140/s vs 120/s capacity
rho = 0.8: drained after  239.9 s (formula  240.0 s); downstream peak 140/s vs 120/s capacity
rho = 0.9: drained after  539.9 s (formula  540.0 s); downstream peak 140/s vs 120/s capacity
rho = 0.8, dropping items older than 30 s: drained after 119.5 s

macOS 26.6.2 · Python 3.13.3 · run 2026-10-20

What we found

The drain times matched the formula: 59.9 s, 240 s and 540 s. During every catch-up the downstream stage received 140 jobs per second against its capacity of 120, and was overloaded for as long as the drain lasted. Dropping items older than 30 s cut the drain at ρ = 0.8 to 119 s.

Read it

Recovery time is a utilisation problem. A system run at 90% takes nine times its outage to recover, and its catch-up is a load spike for every stage after it, which must either have headroom for the catch-up rate or be protected by a rate limit on the replay. Dropping stale work is the only lever that shortens recovery without adding capacity, and it is only legitimate when old items have lost their value (a stale price, an expired session).