Summary and key takeaways
The inversion. Time stops being a cost to minimize and becomes metadata. Correctness means output that does not depend on how fast the pipeline ran. Any leak of execution speed into the answer is a bug.
The core constraint. Completeness is unobservable, because silence is ambiguous. So every time-based result is a bet. The three ways to pay: wait (latency), guess (accuracy), revise (consumers must accept edits).
Three clocks, one non-clock
| Clock | Property |
|---|---|
| Event time | Meaningful, unverifiable (device clocks are adversarial) |
| Processing time | Always present, meaningless, the only one that breaks reproducibility |
| Ingestion time | Trustworthy and replayable, but turns your outages into facts about the world |
| Log position (offset) | No clock; gap-free order; drives progress, de-duplication and replay |
The skew distribution (processing − event) has a median near your latency and a tail 10⁴–10⁵ times larger. Every waiting decision is a percentile choice.
Watermarks. A declared lower bound on future event times. Perfect only if the source proves its order; otherwise max observed − δ, where δ is a quantile (each extra nine multiplies the wait). Across partitions it is a min. Idle sources need a timeout, which is a conscious trade of correctness for liveness and needs an alarm. Watch watermark lag and event-time lag separately.
Late data. Drop (count it), divert, or re-fire. Allowed lateness sets state retention: retained windows ≈ 1 + lateness / windowSize. Exactly-once is not completeness.
Disorder. Bounded disorder K costs O(K) memory. Commutative, associative operators need no buffer at all.
Triggers. Early, on-time, late. Accumulation mode (total vs delta) must match the sink: keyed overwrite or additive.
Partitioning. Event-time partitions turn a query problem into a maintenance problem; ingestion-time partitions do the opposite. Use both, and publish the freeze horizon.
Revision. Output edits need a key (retraction pairs or upserts). Computation revision needs an inverse: sum and count have one; max, min, exact distinct and exact percentiles do not.
Bitemporality. Valid time and transaction time are independent. A correction is a new rectangle, not an update. Type-2 SCD is only the valid-time half. Time travel is only per-table and only transaction time.
Temporal joins. Say as of when. Join against the version valid at the event time, or you leak the future. The correct plan is more expensive than the wrong one.
Calendars and clocks. Store the instant plus the zone name. Local time has duplicates and gaps. Calendar units are not durations. Durations use the monotonic clock. A run's logical time is an input, never an ambient read.
TTL. Expiry redefines guarantees: exactly-once exists only within a de-duplication horizon. Keep dedup TTL ≥ replay/retry horizon ≥ allowed lateness ≥ watermark δ.
Formulas and rules worth remembering:
| Formula | Meaning |
|---|---|
| watermark = max observed event time − δ | Heuristic watermark; δ is a skew quantile |
| global watermark = min over partitions | One slow input freezes everyone |
| retained windows ≈ 1 + lateness / window size | Cost of completeness in state |
| interval-join memory ∝ w × combined rate | Window width is both meaning and memory |
| dedup TTL ≥ replay horizon ≥ lateness ≥ δ | The chain nobody checks |
Patterns that repeat:
- You cannot observe completeness; you declare it and price the error.
- Costs are conserved: latency, accuracy, or the consumer pays.
- Retained state is the one currency: lateness, replay depth, time-travel horizon and TTL are one knob.
- Collapsing two independent axes into one loses information at write time, permanently.
- The cheap plan is often the wrong one (current-version joins, ingestion-time partitions, wall-clock timeouts).