Streaming removes the end

Why does volume stop being the unit of capacity, and why must state outlive the process?

Everything in a batch system assumes a job with an end. A query plan is correct relative to a fixed input. A memory manager spills to disk (writes data to disk when memory is full) because the job is too big, but the spill ends when the job ends. A failure is repaired by running the job again, because there is something to run again.

Streaming removes the end. That single change spreads further than it looks.

The constraint that stops mattering: total input size

Nobody sizes a streaming job by how many terabytes the topic has carried. The quantities that matter are the arrival rate (records per second) and the resident state (bytes the job holds in memory or on local disk). Both are independent of cumulative volume.

• A pipeline that processes a petabyte a year can hold 40 MB of state.
• A pipeline that processes a gigabyte a day can hold a terabyte.

Volume stops being the unit of capacity. (Ana sized her cluster by volume. She measured the wrong thing.)

The constraint that appears: state must outlive the process

In batch, a hash aggregation table (a table that holds one running total per key) is scratch. It is built, drained and thrown away inside one job. In streaming there is no drain point. The same table becomes a long-lived, durable, growing object. It must be:

• checkpointed (saved periodically),
• recovered after a crash,
• size-bounded,
• garbage collected (old entries removed),
• redistributed when the parallelism changes,
• migrated when the code changes.

So the execution engine is forced to grow a storage engine inside itself. It inherits every problem of storage engines: layout, durability, compaction, eviction and cleanup. These problems now live inside what used to be a simple query operator.

Two further consequences

Output becomes a sequence, not a value. Results must be emitted before the input is complete, so any emitted result may turn out to be wrong later. Every streaming system must pick a position on one axis:

• Wait until you believe the input is complete, and emit once. The cost is latency.
• Emit early and revise. The cost is complexity, and it is pushed onto every downstream consumer.

Which result is correct to emit, and when, is the topic of event-time semantics. This mechanism covers the machinery that holds data while you wait, and the "changelog" plumbing that carries revisions when you do not wait.

Planning is replaced by regulation. A batch optimizer sizes a job against known data sizes. A streaming job cannot do that. The arrival rate is an input you do not control, and it changes after deployment. There is also no re-planning, because the plan is coupled to a state layout that you cannot abandon. So the runtime control mechanism is not a cost model but a feedback loop (backpressure), and the deployment mechanism is not recompilation but migration (moving state to a new shape).