Watermarks: declaring completeness
How can a pipeline declare that everything before time T has arrived when it cannot know?
Why a watermark is needed
You want to emit a result for a window and then stop working on it, so that you can free its state. To stop, you need the claim:
You cannot derive this claim from the data you observe, because not having seen a record is not evidence that none exists (absence of an event is not absence of a failure). So the claim must be asserted by the system. This assertion is a watermark:
It is monotone non-decreasing by construction. A watermark that could move backwards would be useless as a licence to release state, because you could never be sure that a released window would stay released.
Analogy. A teacher says "I will collect the exam papers at 11:00. Anyone who has not handed one in by then is late." The teacher cannot know whether a student is still writing. The rule is a declared deadline, not a measurement.
Perfect watermarks
A perfect watermark exists only when the source can prove its own order:
- replaying a sorted file;
- reading a single-writer log whose timestamps are in order;
- a producer that sends explicit progress markers ("nothing older than 10:07 from me").
Where it exists, the completeness problem simply disappears. That is why batch jobs and replays of finished data feel easy, and live workloads feel hard.
Heuristic watermarks
A heuristic watermark is:
- a quantile of the skew distribution
The whole design is in choosing δ (delta). It is a quantile of the skew distribution, so choosing δ is choosing your error rate directly:
- δ at p99 means you accept that 1% of records will be late.
- δ at p99.99 accepts 0.01% late records, but it costs you the p99.99 of a heavy tail. That is not "0.99% more latency", it is often 100× more latency. Each extra "nine" is bought multiplicatively.
Better generators do not use one fixed δ:
- They track the quantile adaptively, for example with a t-digest (a compact structure that estimates quantiles from a stream of numbers) over the observed skew. This is a control loop that uses a sketch.
- They keep δ per source instead of one global value.
The best option is source-declared progress, where the producer promises how far it has got. It requires the producer to accept a contract. The obstacle is organizational, not technical: someone else's team must agree to something.
Several partitions: the watermark is a min
A job reading many partitions keeps one watermark per partition. To get the watermark of the whole stream, it takes the minimum over the partitions. This is the safe choice, because a window can close only when every input has passed it.
That is an AND-coupling: all inputs must be ready. One lagging input freezes the global watermark. Then nothing fires. Then state piles up in operators that are not the broken ones. The memory alarm rings far away from the cause. This is the same arithmetic as tail latency in a fan-out (a request that waits for 100 nodes is as slow as the slowest) and as gang scheduling (a job that needs all its machines at once). The strength of the coupling is the ratio slowest/fastest.
The global watermark = min(11:04, 11:03, 09:19) = 09:19. The 10:00–11:00 window can close only when the watermark passes 11:00, so it waits for partition 7, which will never move.
Idle sources
The pathological case is the idle source: a silent partition never advances its maximum observed time, so the minimum never advances, and the pipeline stalls permanently. This is Kavya's problem.
The fix is an idleness timeout (the same rule inside a job is idle channels): "if a partition has sent nothing for X minutes, ignore it when computing the minimum." It is worth being explicit about what this fix does:
It takes an ambiguous silence and resolves it, by decree, to the harmless explanation: "no data means nothing happened."
If the source was in fact broken and not idle, the watermark moves past data that still exists. That data will then arrive late, or be dropped. The idleness timeout is a deliberate trade of correctness for liveness (keeping the pipeline moving). It is the one configuration value here that most deserves an alarm attached to it.
Watermark generation and two kinds of lag
For operations you need to see two lags, and you must keep them apart.
- Watermark lag = now − watermark. It tells you how stale your claim is.
- Event-time lag = now − largest observed event time. It tells you how stale your input is.
Their difference is δ plus your processing delay. During a backlog replay (catching up after an outage) both lags look terrible while the system is perfectly healthy. This is why freshness SLAs defined on processing time raise false alarms exactly during catch-up. Define freshness on the watermark and the input, and know which one your alert watches.