Freshness
Why is freshness a property of the data rather than the job, and why is a data SLA's arithmetic so coarse?
Freshness = now − max(event_time present in the dataset). Take the newest event timestamp inside the table and subtract it from the current time.
SELECT now() - max(event_time) AS freshness
FROM daily_revenue_source;The choice of clock is load-bearing:
- measured against event time (when the thing happened in the world), freshness includes upstream delay. It is the number consumers actually feel.
- measured against ingestion time (when your system received it), it measures only your segment of the path, and will read green while the source is dead.
The classic false green in this field is measuring "time since last successful run". A job that runs successfully over empty input reports healthy. (This is Nadia's problem.)
Data SLAs have coarse arithmetic
A service fails per request, so "99.9%" means one request in a thousand. A daily dataset has only 365 opportunities per year. One missed day is 364/365 = 99.7%. After one bad Tuesday you are already below 99.9% for the year.
This is why data SLAs are written as "available by 09:00 on 99% of business days" and not as a fraction of wall-clock time, and why service uptime and data uptime are not comparable quantities.
Absence needs an expectation
Nothing emits an event for the file that did not arrive, so every absence detector is a scheduled check against a declared expectation, and the expectation can go out of date. Detecting absence is orchestration's subject; freshness is the signal it reads.