State TTL and expiry

How does state expire, and why is a TTL a silent correctness change?

Unbounded state is always a missing or unenforceable retention rule, so every backend grows expiry machinery. TTL (time to live) means "delete this entry when it has not been touched, or has existed, for a set time." The mechanisms trade thoroughness against cost:

Mechanism	How it works	Cost	Weakness
Lazy cleanup on access	check expiry when an entry is read	free	never touches keys that are never read again, which are exactly the leaking ones
Background incremental scan	scan a slice of state in each cycle	bounded per cycle	a continuous tax on CPU and I/O
Cleanup during compaction	drop expired entries while compaction rewrites files	cheapest thorough option, as it rides on work already scheduled	only as often as compaction runs
Filter during snapshot	skip expired entries when writing a checkpoint	small	state in memory is not reduced

Cleanup during compaction is the same tombstone-collection trick that storage engines use, relocated into the runtime.

The semantics matter more than the mechanism

TTL on the last write expires keys that are updated rarely but still read often.
TTL on the last access keeps "zombie" keys alive forever, because something touches them periodically.
Event-time TTL is the correct one and depends on watermark progress. So an idle or lagging source stops cleanup exactly when you need it.

TTL is a silent correctness change

This is the important property. A join whose state expires below the real time skew between the two streams produces missing rows, not errors. Nothing in the system distinguishes "no match existed" from "the match was evicted." The job assumes that real-world event skew stays inside the configured retention. It cannot verify that assumption. When it breaks, the cost lands on a downstream consumer who sees a slightly smaller number and has no signal at all.

Case, resolvedConstructed case · Ana at Ridelo

One hot restaurant, and the job keeps freezing

With a 30 GB heap per worker the fraud job froze every few minutes, one restaurant's key overloaded a single instance, and state still held users not seen for two years. Where it started: State →

The freezes. Her live heap of 30 GB made every GC pass expensive. She moves the fraud state to an embedded LSM backend on local SSD. Pauses disappear. She now watches compaction backlog, because write stalls would show up as backpressure.
The hot restaurant. She replaces the per-restaurant count with two-phase aggregation: salt with 8 sub-keys, count locally, then combine. The count is a sum, so this is valid. Throughput scales again. She notes that this trick would not work for the order-sensitive fraud pattern in complex event processing.
The two-year-old users. She adds a TTL of 30 days on the last write of each user's record, using cleanup during compaction. She also writes in the design document what the TTL means: a user silent for 30 days is treated as new. That is a business decision, not a memory setting.