State backends: which problems you import
Which family of problems does each state backend import into your job?
The state backend is the component that stores operator state. Choosing one is not a performance knob. It is a decision about which family of storage problems becomes yours.
On-heap objects
State is kept as ordinary objects in the program's memory. Access takes nanoseconds and needs no serialization. But GC cost scales with the live set, and for a streaming job the live set is the entire state. A 30 GB live heap is a GC death spiral waiting for a load spike: the collector must keep examining 30 GB, long pauses make the job fall behind, a backlog builds, more objects are alive, and pauses get longer. Heap size also caps state size at one machine's memory. (This is Ana's freezes.)
Off-heap serialized state
State is stored as bytes outside the garbage-collected heap. GC pressure disappears, and you pay serialization on every access. That per-access fixed cost immediately implies a batch size: you stop doing one lookup per record and start batching lookups, or you keep a small deserialized cache in front. The fixed cost migrated upward, from the disk into the object boundary. That is exactly what the fixed-cost law predicts happens as the layer below gets faster.
Embedded LSM state on local disk
State lives in an LSM-tree (such as RocksDB) on each worker's local disk. This decouples state size from memory: state can be many times larger than RAM. In exchange you inherit the problems of storage engines wholesale:
- write amplification (each record is written several times by compaction),
- compaction debt (compaction work that has fallen behind),
- read amplification (a read may check several files), mitigated by Bloom filters,
- a background compaction thread competing with your processing thread for the same disk and CPU.
Compaction debt under sustained write load is a real streaming failure mode, not a footnote about storage engines. Write stalls in the state backend appear as backpressure in the topology.
Remote or disaggregated state
State lives in object storage or a shared service. This severs state size from node size and makes rescaling nearly free, since a new instance only needs to be told where state is. It pays a round trip (RTT) per access, which forces caching and batched asynchronous access. Now you have the problem of cache coherence (keeping a cache consistent with the real data) inside your operator.
Whether this approach can match local state for tight point-access loops is genuinely unsettled right now. It clearly cannot for the highest-frequency cases. The rescaling and checkpointing wins are large enough that several systems are moving this way anyway. Treat it as current engineering reality in flux, not settled design.
Summary table
| Backend | Where state lives | Access cost | What you inherit | Typical dominant failure |
|---|---|---|---|---|
| On-heap | process memory | ~ns | GC scaling with live set | GC pauses |
| Off-heap | memory outside GC | serialization per access | batching, small cache | CPU in serialization |
| Embedded LSM | local disk | disk + serialization | write/read amplification, compaction | compaction debt, write stalls |
| Remote | object store / service | RTT per access | cache coherence, async batching | network latency |