Watermark propagation

How does a watermark travel through a job, and why can one idle input stall everything?

How a watermark is generated is part of event-time semantics. How it travels through the job is an execution mechanism, and it decides when timers fire.

The rule: the minimum across inputs

An operator may have several input channels (from several upstream instances, or from several source partitions). Each channel carries its own watermark. The operator can promise "nothing older than T will arrive" only if every input promises it. So:

The operator tracks one watermark per channel. When one channel's watermark advances but another's does not, the minimum, and therefore the operator's own watermark, does not move.

At a keyBy

After a keyBy (a shuffle), every downstream instance receives records from every upstream instance. So every downstream instance has as many input channels as there are upstream instances, and its watermark is the minimum over all of them. One slow upstream instance holds back the watermark of all downstream instances.

Idle channels

A channel that has no data (for example an empty source partition at night) never advances its watermark, so it holds the minimum back and no window anywhere closes: the "single quiet partition" in Ana's problem. Engines mark such a channel idle after a configured time and exclude it from the minimum, which declares that silence means nothing happened, with the risk that a resumed channel's events arrive late (idle sources).

The independence assumption

The minimum rule assumes sources advance independently and at similar speeds. In real systems they do not. Partitions differ in traffic, brokers differ in lag, and clocks differ. This is the "independence is assumed, rarely true" law, and it is why the watermark is a bottleneck that is easy to overlook.