SLAs and absence detection
Why can an orchestrator not detect its own absence, and what makes a deadline a service level?
SLAs, deadline alerts and absence detection
Every other mechanism so far is a safety property: nothing bad happens. An SLA (service-level agreement: a promise about timing, such as "the table is fresh by 07:00") is a liveness property: something good eventually happens. Liveness cannot be verified by the system responsible for delivering it. That one sentence generates the whole section.
The run that never started
The failure to guard against is the run that never started. It produces no logs, no failure, no metrics and no exit code. It leaves no evidence of any kind, because nothing ran. You cannot detect it by watching events, for the same reason that completeness could not be detected from arrivals: absence is not an event. To detect a non-event you need an independent clock that asserts: "by now, X should have happened."
Why the orchestrator cannot watch itself
The consequence is usually missed: SLA monitoring inside the orchestrator is structurally unsound for the case that matters most. If the scheduler is down, the runs do not fire and the SLA checks do not fire either. The failure with the largest blast radius is exactly the one that makes no alert.
The correct construction is an external dead-man's switch: an independent process that expects a heartbeat by a deadline and alerts when it does not arrive. (The name comes from train and machine controls that stop when the operator lets go.) The orchestrator's built-in SLA feature covers the case where the orchestrator is healthy and one task is slow. That is useful, but it is the easy case.
Task SLA versus data SLA
A second distinction is usually blurred: the SLA on the task and the SLA on the data are different promises, and only the second one is real. Consumers do not care whether your task finished at 06:00. They care whether the table contains yesterday's data. A task that completes successfully on empty input meets its task SLA and silently misses its data SLA. Measuring data freshness is a monitoring job. The orchestration job is only the deadline semantics and the escalation. Keep them separate. "The pipeline is green and the data is stale" is the most common form of this complaint.
An SLA implies an action
A deadline that only sends an email is a notification, not a service level. What makes it an SLA is what happens downstream when the deadline is missed:
- serve the last known good partition,
- show a staleness banner on the dashboard, or
- block the publish.
This is again fail-fast versus fail-degraded: decide in advance whether a missed deadline blocks or degrades (serves stale data with a marker). The answer differs per consumer, which is why the SLA belongs on the dataset rather than the task.
Missing work emits nothing, so detecting it requires an observer outside the system that was supposed to do it — and the promise consumers actually hold is on the data's freshness, not the task's completion.