The governing question and its five answers
What must be remembered, for how long and where, and why is unbounded state always a missing retention rule?
Given input that never ends: what must be remembered, for how long, where does it live, and what happens to it when something moves?
Everything here is one of five answers to that question.
| # | The answer | What it covers | Where |
|---|---|---|---|
| 1 | What is remembered | stateless and stateful operators, keyed and operator state, windows, joins, pattern automata | State, Time-shaped state and Joins and enrichment |
| 2 | How it survives | state backends, checkpointing, exactly-once | State and Checkpoints and exactly-once |
| 3 | How it moves | keyBy, key groups, rescaling, redistribution, upgrades | Rescaling and upgrades |
| 4 | How the flow is regulated | backpressure through the topology, micro-batch versus record-at-a-time | Backpressure and batching |
| 5 | What the output is | changelog streams, table-stream duality, queryable state | Changelogs and queryable state |
Answer 1 deserves a longer note, because it unifies a lot. Stateless operators, keyed state, windows, joins and pattern matching look like five different features. They are one question wearing five costumes: what is the retention rule for the buffer this operator needs?
- A tumbling window is a retention rule keyed on time.
- An interval join is a retention rule keyed on the allowed time difference between the two sides.
- A pattern automaton is a retention rule keyed on how long a partial match stays possible.
Unbounded state is always a missing, or unenforceable, retention rule. Hold on to that sentence. It explains Ana's Sunday night.
Answer 2: the recovery story is forced. You cannot re-run from the beginning, because the beginning may be years ago. So you must restore the state plus an input position, and that requires the source to be a replayable log. The dependence of streaming on an append-only log is not a style preference. It is the only structure that makes incremental recovery possible.
Answer 3: state is partitioned by key. So changing the parallelism means physically moving state. An indirection layer called key groups exists so that we do not have to move state key by key.
Answer 4: two views of one trade. Backpressure and the batching model are the same trade: how much work do you commit to before you can react?
Answer 5: the duality claim. A table is a materialized changelog, and a changelog is the derivative of a table. This lets streaming output be a first-class input to other jobs. It is also why retractions (messages that take back an earlier result) are unavoidable rather than a design flourish.