Anomaly detection and alert fatigue
Where does anomaly detection earn its keep, and how is alert fatigue suppressed?
Anomaly detection
Replacing a constant threshold with a learned one buys tolerance of daily and weekly patterns, and costs explainability plus a new failure: a model trained on a broken state learns it as normal. For checks on data (volume, freshness, null rate) that trade, and where it earns its keep, is data quality's subject. For the pager, the point is that a learned threshold still produces alerts at a base rate, so everything below applies to it unchanged.
Fatigue and suppression
Alert fatigue is the same fixed-rate-meets-growth law with a human as the fixed resource: alert volume grows with system size, attention does not.
The mechanisms are all suppression:
- Deduplication: many identical alerts become one.
- Grouping by dependency: one root cause yields one page. This requires the lineage graph (the map of which table feeds which). So alert quality is downstream of metadata quality. If your lineage is wrong, one failed source produces two hundred pages. (This is the independence assumption failing: alerts are not independent; they share one cause.)
- Maintenance suppression: silence alerts for work you planned.
- Routing by ownership: each alert goes to the team that owns the thing.
The operational test is severity-to-action. If the response to an alert is "look tomorrow", it is not a page. And every page that resolves itself trains the responder to ignore the next one.