Late data: drop, divert or re-fire
What can be done with a record that arrives after the watermark, and what does accepting it cost in state?
A late record is one whose event time is below the current watermark. Since the watermark was a guess with a chosen error rate (δ at some percentile), lateness is not an anomaly. It is the residual you already budgeted for. If you chose p99.9, you expected 0.1% of records to be late.
There are exactly three things you can do with a late record:
| Disposition | What it means | What it needs |
|---|---|---|
| Drop | Ignore it | Must be counted |
| Divert | Send it to a side output, a separate place for records that did not fit the main path (the same idea as a dead-letter queue: a holding area for records that could not be processed normally) | A place and a person to look at it |
| Re-fire | Update the result of the window | The window's state must still exist |
Allowed lateness sets a retention floor
To re-fire, the state of the window must still be there. So allowed lateness (how long after the window ends you still accept updates) fixes a minimum retention:
State for a window must survive until watermark > windowEnd + allowedLateness.
The number of windows you hold at the same time is:
So for short windows, doubling the allowed lateness roughly doubles the memory. Completeness has a computable price in bytes. It is also the state that must move before a rescaled operator can help: elasticity is bounded by the state the horizon traps.
Silent drops
Dropping a late record is defensible only if it is counted. A silent drop turns a correctness failure into a quiet shift in the distribution of your numbers. It is an instance of the rule "masking moves a failure into a channel that has no schema": the error is still there, but there is no column in which anyone can see it. A pipeline should expose late_records_dropped_total and someone should watch it.
A correction that vendors blur
Kavya's pipeline was exactly-once and still wrong by 160 deliveries (3,260 − 3,100).