Decide what may page you
Alerting is a base-rate problem, not a detection problem. With a real incident in one check in a thousand and a detector that never misses but fires falsely 1% of the time, nine of ten pages are noise, and no better detector fixes that. Human attention has a fixed rate; the system keeps growing. So the design question is what not to say. What may page someone on this platform?
You will end with a short list of pages, each a consumer-felt symptom with an objective, a burn-rate rule and an owner; everything else on dashboards.
Work through these in order.
Work through it
Page on symptoms, not causes
List what consumers feel: "the revenue table is more than 3 hours stale", not "disk at 82%". The only structural fix for the base rate is fewer things that can page. Causes belong on dashboards; symptoms belong in pagers.
Measure freshness from the data, against event time
Freshness is now minus the newest event time present in the dataset. Measured against ingestion time it reads green while the source is dead; "time since the last successful run" reads green when a job succeeds over empty input. Freshness is a property of the data, never of the job.
Write each objective per opportunity
A daily dataset has 365 chances a year to fail: one missed day is already 99.7%. Write objectives the way they fail, such as "available by 09:00 on 99% of business days", not as a fraction of wall-clock time.
Alert on burn rate, and price the rule
A 99.9% monthly objective allows 43.2 minutes of failure. A rule of 14.4× burn over 1 hour fires after 2% of the budget (fast, noisier); 6× over 6 hours after 5% (slower, quieter). Add a short confirmation window, such as the last 5 minutes, so the alert stops when the problem has.
Give every absence an expectation
Nothing emits an event for the file that did not arrive, so every absence detector is a scheduled check against a declared expectation, and the expectation can go out of date.
Suppress before you send
Deduplicate identical alerts, group them by dependency so one root cause gives one page, silence planned work, and route each alert to the team that owns the thing. Grouping needs the lineage graph: if lineage is wrong, one failed source produces two hundred pages.
Apply the severity-to-action test
If the response is "look tomorrow", it is not a page. Every page that resolves itself trains the responder to ignore the next one.
Watch the watcher from outside
An observer that fails together with what it watches produces silence exactly when you need a signal, and silence looks like health. Run the check that the alerting path itself is alive on something that does not share its machines and network.