TTL and expiry

What does expiring state silently change about a guarantee, and which chain of horizons must hold?

With unbounded input and state per key, memory is unbounded. Time is the only dimension along which forgetting can be justified, because "nothing seen since T" is the only available evidence that a key no longer matters. So TTL (time-to-live) is the collector of this mechanism, the second half of "immutable storage plus a collector".

But TTL is a semantic statement dressed as a resource control. When a key expires, any future event for that key is treated as the first one ever seen. For de-duplication state this changes the guarantee:

Every real system offers the qualified version. Most documentation leaves out the qualifier.

The invariant that holds this mechanism together

DefinitionThe expiry invariant →

dedup TTL \geq replay horizon \geq allowed lateness \geq δ

Principle: Guarantees have scope →

Break the chain anywhere, and reprocessing produces duplicates or drops. These four values live in four different configuration files, owned by different people, and nothing in any system checks the chain. This is the typical organizational failure of this mechanism.

The chain requires dedup TTL ≥ 48 hours. It is 2 hours, so a replay of data older than 2 hours is no longer recognized as "already seen" and is counted again. Fix: set the dedup TTL to at least 48 hours (and the rest of the chain, 48 h ≥ 1 h ≥ 5 min, holds).

How expiry is done

Expiry is eager (a timer per key), lazy (checked on access, so untouched keys leak) or done in bulk during a scan or compaction (expiry mechanics). Storage-level TTL is almost always the bulk kind, so expired data is still readable until the next pass: as a compliance deletion guarantee it is weaker than it looks.

Case, resolvedConstructed case · Kavya at Kitebox Deliveries

Parcels delivered twice on the last Sunday of October

On the clock-change Sunday a local-time key got duplicates, a report's date arithmetic depended on the order of operations, a 30-second timeout waited almost an hour after a clock correction, and a replay after an outage counted deliveries twice. Where it started: Calendars, clocks and expiry →

Duplicates and gaps once a year. She stores the instant (UTC) and the zone name Europe/Berlin, and uses the UTC instant in all keys. Local time is computed only for display.
Order of date arithmetic. She writes the rule for the report ("add the month first, then the day") in one function and tests it with the month-end dates.
The long timeout. She replaces time.time() by time.monotonic() for all timeouts and retry delays.
Replay differences. She removes current_timestamp from the transformations and passes run_date as a parameter. Her backfill of last week now equals the original run, and she adds a test that re-runs a day and compares the results.
Duplicates after replay. She sets dedup TTL to 72 hours (above the 48-hour replay horizon) and adds a startup check that fails if the four values violate the chain.

Setting	Value
Watermark δ	5 minutes
Allowed lateness	1 hour
Broker replay horizon (retention)	48 hours
De-duplication TTL	2 hours