What streaming forbids, and what the escapes cost
What can a streaming design never verify?
What this mechanism forbids
- Unbounded retention with bounded state. Every operator needs a retention rule. The operators that appear not to have one have an implicit one, which will be enforced by an out-of-memory crash.
- Exactly-once effects at latency below the checkpoint interval, unless the sink is idempotent.
- Global ordering with parallelism. Order exists per key or not at all.
- Free rescaling of anything stateful.
- Splitting a single hot key.
- Final results before the watermark permits, or revisable results without imposing changelog handling on every consumer downstream.
- Throughput above the slowest operator, with no queueing rescue because queues are bounded by design.
- Code changes without a state migration.
The escapes and what each costs
| Forbidden | Escape | Price |
|---|---|---|
| Unbounded join state | Interval bound, TTL | Silent wrong answers when real skew exceeds the bound |
| Sliding-window S/s blowup | Pane decomposition | Aggregate must be mergeable; exact distinct and median need sketches |
| Hot key caps throughput | Salt + two-phase aggregation | Associative aggregates only; extra shuffle and latency stage |
| Full snapshots too large | Incremental checkpoints | Reference counting, file GC, periodic consolidation, longer restore chains |
| Alignment stalls under load | Unaligned checkpoints | Larger checkpoints, more write bandwidth |
| Lookup RTT caps throughput | Async I/O + cache | Non-determinism, ordering choice, cache staleness |
| Non-reproducible enrichment | Temporal join on a dimension changelog | Multi-version state, second watermark, idle-source stalls |
| 2PC sink fragility | Idempotent writes on deterministic keys | Deterministic key derivation, upsert-capable sink |
| Rescale downtime | Disaggregated state | RTT per access, cache coherence, unsettled for tight loops |
| Code change breaks state | Parallel run + cutover | Double compute, source retention window, idempotent sink |
| State too large for memory | Embedded LSM | Write amplification, compaction debt competing with processing |
Look at the right-hand column: many entries are silence. The characteristic failure of streaming is not a crash. It is a correct-looking number that is slightly wrong, produced by a bound that was set for resource reasons and then quietly became a correctness boundary.
What the design assumes it cannot verify
Three things. Carry them forward as a checklist for any streaming design:
- The source's retention exceeds the worst-case downtime plus catch-up. (Otherwise the log has deleted the data you need to replay.)
- The job's computation is deterministic under replay.
- Real-world event skew stays inside the configured retention bounds.
All three fail silently. In all three cases, the cost lands on a downstream consumer who has no way to distinguish a correct answer from the one they got.