Scenarios

Group by event time; keep ingestion time as a second column; validate event times with plausibility bounds, counting rejects.

Measure the skew: say a median of 2 seconds and a 99.9th percentile of 5 minutes.

Latest event time seen minus 5 minutes, per partition, taking the minimum across partitions, with an idleness timeout and an alarm on it.

Allow 1 hour of lateness (about 1 + 60/60 = 2 hourly windows of state per key); divert anything later to a side table and count it.

Keyed upserts by (depot, hour), labelled provisional until final. A raw zone partitioned by ingestion date; a curated zone by event date, open for 3 days and then frozen, with a published "final after" date.

The live number arrives early and may change; finance reads only frozen partitions; records later than 3 days become labelled adjustments. Watch for an idle partition hiding a real outage, and a sink that does not match the trigger's accumulation mode.

A frozen watermark. Check watermark lag and event-time lag per partition: if one partition's latest event time is far behind the others, the minimum is stuck.

Because the job's watermark is the minimum across inputs, one idle or broken input stops every window from firing, and state accumulates in unrelated operators, so the memory alarm fires far from the cause.

An idleness timeout, stating clearly that it treats silence as "nothing happened" (alarm on it, and watch the late-record count); a separate δ per source; or progress declared by the source itself.

Check whether the silent partition is idle or broken. If broken, expect its data to arrive late once the watermark moves on.

Most likely a mismatch between accumulation mode and sink: the job emits the new total each time a window fires again, but the sink adds rows. For example, 3,100 on time and 3,260 after a correction gives 6,360.

Make the sink a keyed overwrite for accumulating output; or emit only the change (+160) to an additive sink that has seen every earlier firing exactly once. Revising a result needs a key either way.

A reconciliation job comparing sink totals with source counts, and telling consumers plainly that early numbers are provisional.

Use time-travel snapshots, if their retention reaches back to 10 March. Note their limits: time travel is per table and covers transaction time only, so it can show what the table said, but not "true on Monday, believed on Tuesday" for each fact.

A bitemporal table with both valid time and transaction time, where a correction closes the old row's transaction period and inserts a new row.

Storage grows with the rate of corrections, and joins become joins on overlapping intervals.

History lost before this change cannot be recovered: it was collapsed to one timestamp when it was written.

Future leakage: the training data joined each event with the current version of dimension features, not the version valid at the event's time.

An as-of join on the event timestamp against versioned feature history: valid_from <= event_time < valid_to. Expect the offline score to fall to its honest value.

Features computed from processing time or now(), which cannot be reproduced.

The correct plan is a sort-merge over both sides and costs more than a hash join on the key; budget for it, and test with a replay comparison.

Find every read of the ambient clock (current_timestamp, processing-time triggers) and every order-dependent operation (first and last, floating-point sums), and pass the data interval in as an input. Add a test that runs the same day twice and compares the output.

Check the chain deduplication TTL ≥ replay horizon ≥ allowed lateness ≥ δ. For example, a 2-hour deduplication TTL against 48 hours of broker retention makes replayed records older than 2 hours look new. Set the TTL to at least 48 hours, say 72, and add a startup check that rejects configurations breaking the chain.

A longer TTL means more deduplication state. And say plainly that the guarantee is exactly-once only within a finite horizon.

The UTC instant and the zone name, never local text or a bare offset.

For keys and ordering: local time repeats an hour once a year and skips one once a year.

Using the current rules database, and plan for updating it, since rules change, sometimes retroactively. Define days with a calendar-aware library, since a day can have 23 or 25 hours, not with 86,400-second arithmetic.

The fall-back and spring-forward days explicitly, and month-end arithmetic.

Reports for past dates may change when the rules database changes, so record which version of the rules each report used.

Buffered records ≈ join window × combined event rate (Little's Law): at 150 events per second and a 600-second window, about 90,000 records. Doubling traffic doubles it.

• Shorten the window, if the business rule allows: the window is both the meaning and the memory, so this changes results and must be agreed.
• Drop unneeded columns before the join, to shrink each record.
• Partition the join by key, spreading state over more machines.
• Replace the interval join with a keyed as-of lookup into a state table, if the logic permits.

Results are complete only once the watermark passes the event time plus the window, so a longer window also delays output. And rescaling stateful operators means moving this state.