The three ways out of incompleteness
What are the only three ways to act on data that may still be incomplete, and what do they all cost?
Which clock to use is settled once, when the timestamp is assigned. The harder question is when you may act on data that may still be incomplete. Watermarks, triggers, allowed lateness and retractions turn that unanswerable question ("is the data complete?") into a declared assumption that can be checked and found false. There are exactly three ways out of incompleteness, and no fourth:
- Wait. Buy completeness with latency.
- Guess. Emit early and accept that you may be wrong.
- Revise. Emit early, correct later, and force every downstream consumer to accept changes and handle repeated updates safely (to be mutable and idempotent).
This is a general rule of design: a cost is never removed, only moved. Here it moves onto latency, onto accuracy, or onto the consumer.
The past in the world cannot change, but your record of it can, and recording that is the last part of the debt (bitemporal data).
All of it is paid for in the same currency: retained state. Allowed lateness, replay depth, time-travel horizon and TTL are the same knob seen from four sides. One parameter sets both how correct you are and how much memory or storage you use, and you cannot tune the two separately. This links three trade-offs that people usually think about apart: exact versus approximate answers, ordered versus parallel processing, and latency versus completeness.