Alert fatigue by arithmetic

Simulate a modelNeeds Python with NumPy, SciPy

Problem

A check with a 0.1% false-alarm rate sounds strict. Run 12,000 of them every day and they fire about 12 times a day on perfectly clean data, so people learn to ignore them. The textbook correction for many tests (Bonferroni: divide the threshold by the number of checks) silences the false alarms, and with them most of the real ones: a defect must now be enormous to clear the stricter bar. Alert fatigue and silent corruption are the same trade, seen from two sides.

Create 12,000 checks, each of which fires with probability 0.001 on perfectly clean random data. Simulate 90 days. Count the alerts per day (about 12). Then apply a Bonferroni correction (fire only if the check's p-value is below 0.001 ÷ 12,000) and add one injected real defect of moderate size per week. Report how many of the real defects each setting catches.

The question. How many alerts a day does the suite raise on clean data, and how many of the weekly real defects does each setting catch?

Predict

With the Bonferroni correction, how many of the 13 real defects does the suite catch?

All 13MostA fewNone

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

alert-fatigue-by-arithmetic.py51 linesDownload

View the code

alert-fatigue-by-arithmetic.py

# /// script
# requires-python = ">=3.10"
# dependencies = ["numpy", "scipy"]
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Alert fatigue by arithmetic.

12,000 checks, each firing with probability 0.001 on clean data, for 90
days. Then a Bonferroni correction (fire only below 0.001 / 12,000), and one
injected real defect of moderate size per week: how many does each setting catch?
"""
import argparse, json
import numpy as np
from scipy.stats import norm

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--checks", type=int, default=12_000, help="Checks in the suite.")
p.add_argument("--alpha", type=float, default=0.001, help="Per-check false-positive rate.")
p.add_argument("--days", type=int, default=90, help="Days simulated.")
p.add_argument("--effect", type=float, default=3.5, help="Size of an injected defect, in standard errors of its check's statistic.")
p.add_argument("--seed", type=int, default=0, help="Random seed.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()
rng = np.random.default_rng(args.seed)

z_plain = norm.isf(args.alpha / 2); z_bonf = norm.isf(args.alpha / args.checks / 2)
defect_days = list(range(3, args.days, 7))
rows, caught = [], {"plain": 0, "bonferroni": 0}
for d in range(args.days):
    z = rng.standard_normal(args.checks)                      # each check's statistic on clean data
    real = d in defect_days
    if real: z[rng.integers(args.checks)] += args.effect * rng.choice([-1, 1])  # one check sees a real defect
    fires_plain = int(np.sum(np.abs(z) > z_plain)); fires_bonf = int(np.sum(np.abs(z) > z_bonf))
    rows.append({"day": d + 1, "alerts_plain": fires_plain, "alerts_bonferroni": fires_bonf})
    if real:
        hit = np.abs(z) > z_plain
        caught["plain"] += bool(np.max(np.abs(z)) > z_plain)
        caught["bonferroni"] += bool(np.max(np.abs(z)) > z_bonf)
summary = {"alerts_per_day_plain": float(np.mean([r["alerts_plain"] for r in rows])),
           "alerts_per_day_bonferroni": float(np.mean([r["alerts_bonferroni"] for r in rows])),
           "defects": len(defect_days), "caught_plain": caught["plain"], "caught_bonferroni": caught["bonferroni"]}
print(f"per-check threshold {args.alpha}: {summary['alerts_per_day_plain']:.1f} alerts/day on clean data; "
      f"caught {caught['plain']} of {len(defect_days)} real defects (each among ~12 false alarms that day)")
print(f"Bonferroni ({args.alpha} / {args.checks:,}): {summary['alerts_per_day_bonferroni']:.2f} alerts/day; "
      f"caught {caught['bonferroni']} of {len(defect_days)} real defects")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"days": rows, "summary": [summary]}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run alert-fatigue-by-arithmetic.py
uv run alert-fatigue-by-arithmetic.py --help     # every knob
```

Knobs
Knob · default	What it changes
--checks12,000	Checks in the suite.
--alpha0.001	Per-check false-positive rate.
--days90	Days simulated.
--effect3.5	Size of an injected defect, in standard errors of its check's statistic.
--seed0	Random seed.

Our run

90 days, one real defect a week

90 days, one real defect a week
Alerts/day, plain	Defects caught, plain	Alerts/day, Bonferroni	Defects caught, Bonferroni	Real defects
12.2	13	0	0	13

Reference runApple M2 · 8 cores · 8 GB

Output · uv run alert-fatigue-by-arithmetic.py

per-check threshold 0.001: 12.2 alerts/day on clean data; caught 13 of 13 real defects (each among ~12 false alarms that day)
Bonferroni (0.001 / 12,000): 0.00 alerts/day; caught 0 of 13 real defects

macOS 26.6.2 · Python 3.13.3 · numpy 2.5.4 · scipy 1.18.1 · run 2026-10-20

What we found

On clean data the suite raised 12.2 alerts a day, as expected. It caught 13 of the 13 real defects, each one buried among that day's false alarms. With the Bonferroni correction it raised 0 alerts a day and caught 0 of the real defects: quiet, and blind to defects of this size.

Read it

Ten thousand monitors, each with a 1% false-alarm rate, produce about a hundred false alarms every day by construction. Neither setting is the answer. Fewer, better-targeted checks on what matters; thresholds in business units rather than p-values; and routing alerts by severity so that a person sees the dozen that matter, not the hundreds that fired.