Event-driven triggering
What do events buy, and why can an arrival stream never say a batch is complete?
Time-based firing assumes that readiness is correlated with the clock. When the upstream arrives irregularly, that assumption costs either latency (schedule late enough to be safe) or correctness (schedule early and process a partial interval). Event triggering removes the guess. It fires on an actual signal: a file landed, a message arrived, a table version advanced.
What it buys is latency proportional to actual readiness, instead of latency proportional to the schedule's safety margin. What it cannot buy is completeness.
The cleanest impossibility in the layer
An arrival event tells you that something arrived. Nothing in the event stream tells you that everything arrived. The evidence for completeness would be the absence of further arrivals, and absence is not observable in finite time. You cannot tell "the source is done" from "the source is slow." This is the watermark problem from stream processing (a watermark is a stream's guess of "no more data older than T will arrive") arriving in the control plane instead of the data plane.
So every event-driven pipeline needs a completeness signal that is not an arrival event. There are only three shapes of it:
| Signal | How it works | Weakness |
|---|---|---|
| Sentinel | The producer writes a marker last (_SUCCESS, or a manifest listing the expected files). | A promise by the producer; unverifiable; lost if the producer crashes between the last file and the marker. |
| Count | A number known in advance is compared with the arrivals. | Needs out-of-band knowledge of the expected count. |
| Timeout | Wait Δ after the last arrival, then declare done. | A heuristic watermark; wrong with some probability that you chose, whether or not you computed it. |
There is no fourth option. If someone says their event-driven pipeline "just knows" when the batch is complete, one of these three is buried inside it.
Events give you latency; they cannot give you completeness, because completeness is an absence and absences are not events.