Base-rate simulation of alerting

Simulate a modelNeeds Python

Problem

An alert's usefulness is its precision: the share of pages that are real. When incidents are rare, even a detector that never misses one produces mostly false pages unless its false-positive rate is far below the incident rate, because false positives are drawn from the vast majority of checks where nothing is wrong. This is the base-rate effect.

Simulate 1,000,000 checks with an incident probability of 10⁻³. Use a detector with recall 1.0 and false-positive rates of 5%, 1%, 0.1% and 0.01%. Count the pages and compute precision for each.

The question. What share of pages are real at each false-positive rate?

Predict

At a 0.01% false-positive rate, ten times below the incident rate, what share of pages are real?

Under half50 to 80%80 to 95%Over 95%

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

base-rate-simulation-of-alerting.py35 linesDownload

View the code

base-rate-simulation-of-alerting.py

# /// script
# requires-python = ">=3.10"
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Base-rate simulation of alerting.

One million checks with an incident probability of 10^-3, and a detector with
recall 1.0 and false-positive rates of 5%, 1%, 0.1% and 0.01%: how many pages,
and what share of them are real (precision)?
"""
import argparse, json, random

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--checks", type=int, default=1_000_000, help="Checks to simulate.")
p.add_argument("--p-incident", type=float, default=1e-3, help="Incident probability per check.")
p.add_argument("--seed", type=int, default=0, help="Random seed.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
random.seed(args.seed)

rows = []
for fpr in (0.05, 0.01, 0.001, 0.0001):
    tp = fp = 0
    for _ in range(args.checks):
        if random.random() < args.p_incident: tp += 1
        elif random.random() < fpr: fp += 1
    rows.append({"fpr_pct": fpr * 100, "pages": tp + fp, "real": tp, "precision_pct": tp / (tp + fp) * 100,
                 "formula_pct": args.p_incident / (args.p_incident + (1 - args.p_incident) * fpr) * 100})
for r in rows: print(f"fpr={r['fpr_pct']:<6g}% pages={r['pages']:>6}  real={r['real']:>5}  precision={r['precision_pct']:.1f}%  (formula {r['formula_pct']:.1f}%)")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"detectors": rows}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run base-rate-simulation-of-alerting.py
uv run base-rate-simulation-of-alerting.py --help     # every knob
```

Knobs
Knob · default	What it changes
--checks1,000,000	Checks to simulate.
--p-incident0.001	Incident probability per check.
--seed0	Random seed.

Our run

One million checks, incident rate 10⁻³

One million checks, incident rate 10⁻³
False-positive rate	Pages	Real	Precision	Formula
5%	51K	1,060	2.08%	1.96%
1%	11K	944	8.6%	9.1%
0.1%	2.04K	998	49%	50%
0.01%	1.1K	983	89.4%	90.9%

Reference runApple M2 · 8 cores · 8 GB

Output · uv run base-rate-simulation-of-alerting.py

fpr=5     % pages= 51022  real= 1063  precision=2.1%  (formula 2.0%)
fpr=1     % pages= 10979  real=  944  precision=8.6%  (formula 9.1%)
fpr=0.1   % pages=  2036  real=  998  precision=49.0%  (formula 50.0%)
fpr=0.01  % pages=  1099  real=  983  precision=89.4%  (formula 90.9%)

macOS 26.6.2 · Python 3.13.3 · run 2026-10-20

What we found

At a 1% false-positive rate, 11K pages were sent and 8.6% were real; at 0.1%, 49%; at 5%, 2.08%. Only at 0.01%, ten times below the incident rate, did most pages become real (89.4%).

Read it

A detector's false-positive rate must be compared with the incident rate, not with zero. When incidents are one in a thousand checks, "99% specific" means nine false pages for every real one, and people learn to ignore the pager. Fewer, better-targeted checks (on symptoms users feel, over windows long enough to average out noise) raise precision more than better thresholds on many checks.