Summary and key takeaways

The reframe. Streaming removes the end of the input. Total volume stops being the capacity unit. State must now outlive the process, so the engine grows a storage engine inside itself. In batch, state is scratch and the answer is the deliverable. In streaming, the answer is scratch and the state is the deliverable.

The governing question. What must be remembered, for how long, where does it live, and what happens to it when something moves? Five answers: what is remembered, how it survives, how it moves, how the flow is regulated, what the output is.

The key arithmetic:

Mechanisms in one line each:

• Stateless operators: free replay, rescale and fusion, until they read something they did not receive.
• Keyed state: one writer per key, so no locks; but a hot key cannot be split.
• State backends: choose which family of problems you import (GC, serialization, compaction, network latency).
• TTL: the defence against unbounded state, and a silent correctness change.
• Windows: retention rules in time; panes make sliding windows cheap only for mergeable aggregates; sessions need a maximum duration.
• Timers: stored state, fired in bulk by a watermark jump.
• Watermark propagation: minimum across all inputs; one idle input stalls everything unless idleness is handled.
• CEP: state is partial matches; pruning decides whether it is bounded.
• Stream-stream joins: always interval joins; errors are absences.
• Enrichment: correct enrichment needs a temporal join on a dimension stream with versions and a watermark.
• Changelogs: a stateful operator's output is a changelog; retraction needs invertible or recomputable aggregates.
• Queryable state: trades every isolation property for one network hop.
• Checkpoints: a consistent cut; aligned versus unaligned moves cost between latency and bandwidth; incremental needs immutable files and garbage collection.
• Exactly-once: exactly-once effect = idempotence + consistent cut; costs latency at the sink; needs deterministic replay.
• keyBy and key groups: an indirection table whose size you must guess on day one.
• Rescaling and upgrades: state migration with a lag penalty; explicit operator IDs and compatible schemas.
• Backpressure: converts memory problems into lag; safe only because the durable log absorbs it.
• Micro-batch versus continuous: both batch; they differ in where the boundary sits, whether it adapts, and whether it is a transaction boundary.

Patterns that repeat:

• No retention rule means an implicit one, enforced by memory.
• An immutable substrate grows an indirection layer and a collector (state files, checkpoint files, compacted changelogs).
• Cost is relocated, not removed (latency to bandwidth, correctness to latency, memory to the retention bill).
• Fixed costs migrate upward (disk to serialization to kernel to scheduling).
• Headroom is the recovery budget.
• Failures are silent: a slightly smaller number, not a crash.

The three unverifiable assumptions (the checklist): source retention exceeds downtime plus catch-up; the computation is deterministic under replay; real skew stays inside the retention bounds.