Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Stateless operators: the part that is still just batch

What do stateless operators get for free, and when does an operator stop being stateless?

A produces output that depends only on the record in hand. For example, filter(order.amount > 0) or map(order -> order.userId). This one property makes it cheap in every dimension that matters here:

  • Recovery state is one offset. After a crash, there is nothing to restore except "where was I in the log?".
  • Rescaling moves nothing. You can run 4 or 40 copies, and there is no state to redistribute.
  • Fusion is free. Consecutive stateless operators can be fused (merged) into one task, so a filter → map → project chain costs one chain of function calls instead of three network hops with at each hop.

Stateless operators are the part of a streaming job that is still just batch.

The boundary condition. "Stateless" means no engine-managed state. It does not mean no side effects.

  • A map that calls a remote service is stateless to the scheduler, but its result can differ when the same record is processed again after a crash (non-deterministic on ). This quietly breaks exactly-once (Exactly-once).
  • A filter whose predicate is updated by a broadcast control stream (a second stream of rule updates sent to every instance) is genuinely stateful. It must be declared stateful, or the predicate silently resets to its default after recovery.