Micro-batch versus record-at-a-time

Where do the two execution models really differ, if both batch?

Two models of execution exist.

Record-at-a-time (continuous): each record flows through the operators individually.
Micro-batch: the engine collects records for a short interval (for example 1 second) into a small batch, and runs a small batch job on it.

The trade is per-record fixed costs versus per-batch fixed costs: the fixed-cost law applied to the scheduling boundary rather than to I/O.

Micro-batch amortizes task scheduling, checkpoint coordination and state access over N records. It also unlocks batch-world optimizations: vectorized execution (processing many values per instruction), whole-stage code generation (compiling a chain of operators into one function), and columnar state access. Its costs: it floors latency at the batch interval, and it makes latency bimodal under load. Example: with a 1-second interval, if a batch takes 1.5 seconds to process, the next batch must queue, and its records wait even longer, so latencies suddenly jump from about 1 s to several seconds.

Record-at-a-time floors latency at per-hop network time plus processing time. It pays per-record overhead everywhere. And it needs continuous checkpointing machinery (barriers, A consistent cut), rather than getting a natural transaction boundary for free.

The folklore correction

The usual belief is "micro-batch means high latency, continuous means low latency". The gap has closed from both ends. Micro-batch intervals fell to sub-second. Continuous engines added internal buffering with flush timeouts (keyBy and key groups), which is micro-batching at the network boundary.

The real remaining difference is not latency in principle but:

where the batching boundary sits, and
whether it is adaptive. A fixed batch interval is a static decision. A buffer timeout is a per-channel adaptive one that degrades to full-buffer batching under load automatically.

And the durable reason micro-batch persisted is not throughput. It is that the batch is a natural transaction boundary. A batch has a clear start and end, so exactly-once and recovery are straightforwardly correct, without barrier alignment and 2PC.

	Micro-batch	Record-at-a-time
Fixed costs paid	per batch	per record (or per buffer)
Latency floor	batch interval	network hop + processing
Under overload	bimodal (batches queue)	gradual (buffers fill, backpressure)
Transaction boundary	natural (the batch)	needs barriers and 2PC
Batch optimizations (vectorization)	easy	harder
Batching boundary	fixed interval	adaptive per channel

Case, resolvedConstructed case · Ana at Ridelo

Four operators red, and one culprit

A slow hour at the billing database turned backpressure red at five operators, and a colleague asked whether switching to micro-batch would be faster. Where it started: Backpressure and batching →

The four red operators. She reads the graph from the sink backwards. The only operator without backpressure is the billing sink, so the sink's database is the bottleneck. She adds a limited-rate write buffer and discusses with the database team a larger write capacity. She tells the team the log is the buffer: lag will grow safely, as long as the topic's retention (7 days) is longer than the worst slow period plus catch-up.
The micro-batch question. She answers that neither is "faster". Her job has a latency target of a few seconds and needs exactly-once for billing, which a batch boundary gives naturally, while her continuous engine already has barriers and idempotent sinks in place. Switching would mean redoing all the state and checkpoint machinery. She decides to keep the current model, and to tune the buffer timeout (lower for the fraud path, higher for the batch-like reporting path).