Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Batching

What does batching cost that people forget?

is the plainest form of the idea that every operation has a fixed cost. You accept extra per record in order to share the fixed cost across many records. (A single machine has the same trade: one disk seek or one cache line load is paid once and spread over many bytes. Here the "seek" is a network .)

Why it works, mechanically

Every operation has a fixed cost that does not depend on the size of the payload: a network round trip (RTT), a system call, an fsync (forcing data onto disk), a lock acquisition, a query plan, a file open, an API authentication check. With batch size B and fixed cost F, the fixed cost per record is .

rises steeply as B goes 1 → 10 → 100. Then it flattens, once F/B is small compared with the marginal cost of each extra record.

The returns diminish, and they diminish fast. Going from 1 to 100 might give a 50× gain. Going from 100 to 10,000 might give 2×, for 100× more latency.

Almost every batch-size tuning conversation takes place in the flat region, where you pay latency for nothing. Find the knee (the point where the curve bends) by measuring, sit just past it, and stop.

Size-based, time-based, both, or adaptive

  • Size-based: flush when there are N records or B bytes. Downstream behavior is predictable (uniform units of work, uniform file sizes, which matters a lot for columnar file formats). But latency is unbounded at low traffic: at 3 a.m., your last 40 records wait in the buffer until morning. Never deploy size-only batching.
  • Time-based: flush every T. Latency is bounded by design. But the size is unbounded under a burst: a 10× traffic spike gives a 10× batch that may exceed a request size limit, blow a memory budget, or time out the downstream write.
  • Both, whichever comes first: the correct default. flush when (size ≥ B) OR (age ≥ T). T is your latency target (your , service level objective, the promise about how fast you respond). B is the unit your downstream is comfortable with. This is what Kafka call linger.ms plus batch.size, what Kafka call fetch.max.wait.ms plus fetch.min.bytes, and what every sensible batching client does.
  • Adaptive: the batch size reacts to observed conditions. The clean version is the one that got right and most reimplementations get wrong:

This tunes itself with no configuration. Under low load you get the minimum latency. Under high load you get the maximum throughput. The change between them is automatic.

Group commit in databases works exactly like this. While one fsync is in progress, arriving transactions wait in a group. When the fsync finishes, all waiting transactions commit together with one more fsync. The batch size emerges from the load. There is no knob.

This pattern generalizes: the best adaptive policies use the downstream's own state as the signal, instead of trying to predict load. Prediction is . Observing the number of operations in flight is trivial.

The costs people forget

  • Failure granularity gets coarser. A batch of 1,000 records fails as one unit. Retrying the whole batch the 999 good records. That is fine if the processing is (repeating it is harmless), and corrupting if it is not. Splitting the batch to find the one bad record needs a path ( the halves, then the halves of the bad half) that you must write yourself. If the sink is only partly transactional, you can also get partial application, which is the worst of both worlds.
  • . The last factor is the one that surprises people during an incident.
  • Latency variance rises. With time-based flush, the first record in a batch waits T and the last waits almost 0. Your p50 is fine and your p99 is T. (p99 means "99% of requests were faster than this".) If you have a latency target, T is your p99. Budget for it.
  • Batching at two layers can multiply delays, not just add them. The classic case is Nagle's algorithm plus , a fixed 40 ms stall: see Zero-copy, Nagle and TLS.

Micro-batching

is not a separate mechanism. It is time-based batching with T so small (about 100 ms to a few seconds) that the system calls itself streaming. Spark Structured Streaming uses it by default, Flink commits at boundaries in a similar way, and so do most lakehouse writers (programs that write table files to object storage).

The reason it dominates in practice: the batch boundary is a natural transaction and checkpoint boundary. You commit , write files and update metadata once per batch instead of once per record. Recovery means "redo the last batch". processing becomes manageable. You get nearly streaming latency with the operational simplicity of batch processing and the good file sizes of batch.

The costs:

  • The latency floor is T plus scheduling overhead.
  • You cannot express true per-event semantics (for example, session windows with fine-grained timers become awkward).
  • There is a stability constraint that people often learn the hard way. If the processing time of a batch exceeds T, batches up and the lag grows without bound. The system looks fine for hours and then stops keeping up quickly (a "cliff"). Monitor batch_processing_time / batch_interval. If it stays above about 0.7, you are already in trouble.

Batching is not windowing

is easy to confuse with batching, so we separate them. Windowing assigns records to computation boundaries (for example "the sum for each 5-minute period") based on event time, the time the thing happened. It must handle late data, (the system's estimate of "events up to this time have all arrived") and allowed-lateness policy. Batching groups records for transfer based on arrival time, and has no idea of late data. One record belongs to exactly one batch, and possibly to many windows (sliding windows, session windows). Windowing belongs with computation and state, not with data movement, so this mechanism does not cover it further.