Multiple comparisons and alert fatigue

Why does a large expectation suite produce alert fatigue by arithmetic?

Multiple comparisons: alert fatigue is arithmetic

This is arithmetic, not a discipline problem. Freshlane has 200 tables × 20 columns × 3 checks = 12,000 assertions. Suppose each assertion has a false-positive rate of one in a thousand per run, and runs once a day. Expected false alarms per day = 12,000 × 0.001 = 12. Every day, forever, even if the data is perfect.

(The textbook name for the underlying effect is the multiple-comparisons problem: test enough hypotheses and some will "fail" by chance.)

Alert fatigue is therefore a designed-in property of any large expectation suite. Every response is lossy:

Response	What you lose
Raise thresholds	recall (more misses)
Correct family-wise (e.g. divide the significance level by the number of tests, the Bonferroni correction)	power (smaller real defects are missed)
Aggregate to one verdict per dataset	locality (you no longer know which column)
Give paging authority only to a handful of checks	coverage where you did not give authority

Every threshold is a classifier and inherits two error types. There is no operating point with neither false alarms nor misses. There is only a choice of which to buy. Alert fatigue and silent corruption are the same dial at opposite ends. A quality system tuned for zero noise is, by construction, tuned for low recall.

Every mature stack converges on the last option: a small set of gates with teeth, and a large set of signals without. That convergence is empirical evidence for the principle stated in Gates, quarantine, stop-the-line.

Compression: statistics turn checks into classifiers. At scale, significance measures your row count, and false alarms are arithmetic.

Case, resolvedConstructed case · Ravi at Freshlane

Alarms at 2 a.m. that are gone by noon

An every-order-has-a-customer check raised 150 alarms each night that cleared by noon, a ±20% volume check fired every Sunday, and 12,000 auto-generated checks buried the channel in alerts. Where it started: Beyond one record →

Orphan orders at 2 a.m. Ravi picks the "allowed lateness" escape. The check now asserts only on orders older than 70 minutes (the 60-minute customer load plus a 10-minute margin). Alarms drop from 150 a night to 1 or 2. He accepts that an orphan created in the last 70 minutes is noticed 70 minutes late.
Sunday alarms. He conditions the volume baseline on the day of week: each day is compared with the same weekday in the last 8 weeks. Sundays stop firing. A Sunday of 100,000 orders now correctly fires (it is 33% below the Sunday baseline).
The empty file. Freshness and volume checks on the partner feed become the first gates: a file with fewer than 50% of the expected row count is not published.
The 12,000 checks. He keeps 40 as gates with paging authority and demotes the rest to dashboard signals. He also requires a one-line reason for every constraint; auto-generated ones go to a review queue as "candidates".