Alerting is a base-rate problem, not a detection problem
Why are nine of ten pages noise even with a perfect detector, and what is the only structural fix?
A false positive is an alert with no real problem. A base rate is how often the real thing happens. Here is the arithmetic.
- Assume a real incident exists at about 10⁻³ per check (one check in a thousand).
- Assume a detector with 100% recall (it never misses a real incident) and 1% false positive rate (on healthy checks it fires 1% of the time).
Per 1,000 checks: 1 real incident (and 1 true alert) and about 999 × 0.01 ≈ 10 false alerts.
RelationshipAlert precision is a base-rate problem →
- how often the real thing happens
- how often the detector fires on healthy checks
1% false positives0.1%
Data table
| Base rate of real incidents | series | Precision |
|---|---|---|
| 0 | 1% false positives | 0.1% |
| 0 | 1% false positives | 0.112% |
| 0 | 1% false positives | 0.126% |
| 0 | 1% false positives | 0.141% |
| 0 | 1% false positives | 0.158% |
| 0 | 1% false positives | 0.178% |
| 0 | 1% false positives | 0.199% |
| 0 | 1% false positives | 0.223% |
| 0 | 1% false positives | 0.251% |
| 0 | 1% false positives | 0.281% |
| 0 | 1% false positives | 0.315% |
| 0 | 1% false positives | 0.354% |
| 0 | 1% false positives | 0.397% |
| 0 | 1% false positives | 0.445% |
| 0 | 1% false positives | 0.499% |
| 0 | 1% false positives | 0.559% |
| 0 | 1% false positives | 0.627% |
| 0 | 1% false positives | 0.703% |
| 0 | 1% false positives | 0.788% |
| 0 | 1% false positives | 0.883% |
| 0 | 1% false positives | 0.99% |
| 0 | 1% false positives | 1.11% |
| 0 | 1% false positives | 1.24% |
| 0 | 1% false positives | 1.39% |
| 0 | 1% false positives | 1.56% |
| 0 | 1% false positives | 1.75% |
| 0 | 1% false positives | 1.96% |
| 0 | 1% false positives | 2.19% |
| 0 | 1% false positives | 2.45% |
| 0 | 1% false positives | 2.74% |
| 0 | 1% false positives | 3.07% |
| 0 | 1% false positives | 3.43% |
| 0 | 1% false positives | 3.83% |
| 0 | 1% false positives | 4.28% |
| 0.001 | 1% false positives | 4.77% |
| 0.001 | 1% false positives | 5.33% |
| 0.001 | 1% false positives | 5.94% |
| 0.001 | 1% false positives | 6.62% |
| 0.001 | 1% false positives | 7.36% |
| 0.001 | 1% false positives | 8.19% |
| 0.001 | 1% false positives | 9.1% |
| 0.001 | 1% false positives | 10.1% |
| 0.001 | 1% false positives | 11.2% |
| 0.001 | 1% false positives | 12.4% |
| 0.002 | 1% false positives | 13.7% |
| 0.002 | 1% false positives | 15.1% |
| 0.002 | 1% false positives | 16.7% |
| 0.002 | 1% false positives | 18.3% |
| 0.003 | 1% false positives | 20.1% |
| 0.003 | 1% false positives | 22% |
| 0.003 | 1% false positives | 24.1% |
| 0.004 | 1% false positives | 26.3% |
| 0.004 | 1% false positives | 28.6% |
| 0.004 | 1% false positives | 31% |
| 0.005 | 1% false positives | 33.5% |
| 0.006 | 1% false positives | 36.1% |
| 0.006 | 1% false positives | 38.8% |
| 0.007 | 1% false positives | 41.6% |
| 0.008 | 1% false positives | 44.5% |
| 0.009 | 1% false positives | 47.3% |
| 0.01 | 1% false positives | 50.3% |
| 0.011 | 1% false positives | 53.2% |
| 0.013 | 1% false positives | 56% |
| 0.014 | 1% false positives | 58.9% |
| 0.016 | 1% false positives | 61.7% |
| 0.018 | 1% false positives | 64.4% |
| 0.02 | 1% false positives | 67.1% |
| 0.022 | 1% false positives | 69.6% |
| 0.025 | 1% false positives | 72% |
| 0.028 | 1% false positives | 74.4% |
| 0.032 | 1% false positives | 76.6% |
| 0.036 | 1% false positives | 78.6% |
| 0.04 | 1% false positives | 80.6% |
| 0.045 | 1% false positives | 82.4% |
| 0.05 | 1% false positives | 84.1% |
| 0.056 | 1% false positives | 85.6% |
| 0.063 | 1% false positives | 87.1% |
| 0.071 | 1% false positives | 88.4% |
| 0.079 | 1% false positives | 89.6% |
| 0.089 | 1% false positives | 90.7% |
| 0.1 | 1% false positives | 91.7% |
| 0 | 0.1% | 0.99% |
| 0 | 0.1% | 1.11% |
| 0 | 0.1% | 1.24% |
| 0 | 0.1% | 1.39% |
| 0 | 0.1% | 1.56% |
| 0 | 0.1% | 1.75% |
| 0 | 0.1% | 1.96% |
| 0 | 0.1% | 2.19% |
| 0 | 0.1% | 2.45% |
| 0 | 0.1% | 2.74% |
| 0 | 0.1% | 3.07% |
| 0 | 0.1% | 3.43% |
| 0 | 0.1% | 3.83% |
| 0 | 0.1% | 4.28% |
| 0 | 0.1% | 4.77% |
| 0 | 0.1% | 5.32% |
| 0 | 0.1% | 5.94% |
| 0 | 0.1% | 6.61% |
| 0 | 0.1% | 7.36% |
| 0 | 0.1% | 8.18% |
| 0 | 0.1% | 9.09% |
| 0 | 0.1% | 10.1% |
| 0 | 0.1% | 11.2% |
| 0 | 0.1% | 12.4% |
| 0 | 0.1% | 13.7% |
| 0 | 0.1% | 15.1% |
| 0 | 0.1% | 16.6% |
| 0 | 0.1% | 18.3% |
| 0 | 0.1% | 20.1% |
| 0 | 0.1% | 22% |
| 0 | 0.1% | 24% |
| 0 | 0.1% | 26.2% |
| 0 | 0.1% | 28.5% |
| 0 | 0.1% | 30.9% |
| 0.001 | 0.1% | 33.4% |
| 0.001 | 0.1% | 36% |
| 0.001 | 0.1% | 38.7% |
| 0.001 | 0.1% | 41.5% |
| 0.001 | 0.1% | 44.3% |
| 0.001 | 0.1% | 47.1% |
| 0.001 | 0.1% | 50% |
| 0.001 | 0.1% | 52.9% |
| 0.001 | 0.1% | 55.8% |
| 0.001 | 0.1% | 58.6% |
| 0.002 | 0.1% | 61.4% |
| 0.002 | 0.1% | 64% |
| 0.002 | 0.1% | 66.7% |
| 0.002 | 0.1% | 69.2% |
| 0.003 | 0.1% | 71.6% |
| 0.003 | 0.1% | 73.9% |
| 0.003 | 0.1% | 76% |
| 0.004 | 0.1% | 78.1% |
| 0.004 | 0.1% | 80% |
| 0.004 | 0.1% | 81.8% |
| 0.005 | 0.1% | 83.4% |
| 0.006 | 0.1% | 85% |
| 0.006 | 0.1% | 86.4% |
| 0.007 | 0.1% | 87.7% |
| 0.008 | 0.1% | 88.9% |
| 0.009 | 0.1% | 90% |
| 0.01 | 0.1% | 91% |
| 0.011 | 0.1% | 91.9% |
| 0.013 | 0.1% | 92.7% |
| 0.014 | 0.1% | 93.5% |
| 0.016 | 0.1% | 94.2% |
| 0.018 | 0.1% | 94.8% |
| 0.02 | 0.1% | 95.3% |
| 0.022 | 0.1% | 95.8% |
| 0.025 | 0.1% | 96.3% |
| 0.028 | 0.1% | 96.7% |
| 0.032 | 0.1% | 97% |
| 0.036 | 0.1% | 97.4% |
| 0.04 | 0.1% | 97.6% |
| 0.045 | 0.1% | 97.9% |
| 0.05 | 0.1% | 98.1% |
| 0.056 | 0.1% | 98.3% |
| 0.063 | 0.1% | 98.5% |
| 0.071 | 0.1% | 98.7% |
| 0.079 | 0.1% | 98.9% |
| 0.089 | 0.1% | 99% |
| 0.1 | 0.1% | 99.1% |
At a one percent false positive rate, nine of ten pages are noise. No improvement in detector quality fixes this, because the term that dominates is the ratio of non-events to events. (Even a detector with a 0.1% false positive rate gives only about 50% precision here.)
The only structural fix is to reduce the number of things you alert on:
"Revenue table is stale for more than 3 hours" is a symptom. "Disk at 82%" is a cause (maybe).