Skip to content
About
Main content

Event-driven triggering

What do events buy, and why can an arrival stream never say a batch is complete?

Time-based firing assumes that readiness is correlated with the clock. When the upstream arrives irregularly, that assumption costs either latency (schedule late enough to be safe) or correctness (schedule early and process a partial interval). Event triggering removes the guess. It fires on an actual signal: a file landed, a message arrived, a table version advanced.

What it buys is latency proportional to actual readiness, instead of latency proportional to the schedule's margin. What it cannot buy is completeness.

The cleanest impossibility in the layer

An arrival event tells you that something arrived. Nothing in the event stream tells you that everything arrived. The evidence for completeness would be the absence of further arrivals, and absence is not observable in finite time. You cannot tell "the source is done" from "the source is slow." This is the problem from stream processing (a watermark is a stream's guess of "no more data older than T will arrive") arriving in the instead of the .

So every event-driven pipeline needs a that is not an arrival event. There are only three shapes of it:

SignalHow it worksWeakness
SentinelThe producer writes a marker last (_SUCCESS, or a manifest listing the expected files).A promise by the producer; unverifiable; lost if the producer crashes between the last file and the marker.
CountA number known in advance is compared with the arrivals.Needs out-of-band knowledge of the expected count.
TimeoutWait Δ after the last arrival, then declare done.A heuristic watermark; wrong with some probability that you chose, whether or not you computed it.

If someone says their event-driven pipeline "just knows" when the batch is complete, one of these three is buried inside it.

Events give you latency; they cannot give you completeness, because completeness is an absence and absences are not events.