Buffering

Why does a bigger buffer not help?

A buffer decouples the producer's rate from the consumer's rate for a bounded amount of work. Think of the waiting room of a clinic: it lets patients arrive in a burst while the doctor works at a steady speed. The number that matters is not the buffer's size in bytes but its capacity in seconds:

The four decisions about any buffer

1. Where does it live?

Moving down the table, durability goes up and speed goes down. That is no surprise. But note one thing: an in-memory buffer is a hidden durability decision. If a record is acknowledged into an in-memory buffer and the process restarts, the record is lost. This is silent data loss, and no metric shows it. If your producer is told "OK" before the buffer is durable, you have chosen at-most-once delivery (a message may be lost but never repeated) without saying so. This is the second half of Ravi's problem: the lost in-flight work.

2. Bounded or unbounded?

Always bounded. An unbounded queue is a latency bomb. It accepts work that it will never process in time, so records reach the consumer already stale and useless. The out-of-memory kill comes anyway, only later and with more work destroyed. The only thing an unbounded buffer buys is a longer wait before the failure, and the failure is total instead of partial.

3. What is the full-policy?

There are four choices:

• Block the producer (this becomes backpressure, below).
• Drop-newest (reject the new arrival; this is load shedding, below).
• Drop-oldest (prefer fresh data over complete data).
• Spill to a bigger, slower place.

Drop-oldest is underused. Use it when a newer record makes an older one worthless: metrics, sensor readings, cache invalidations, "who is online" status, price ticks. Dropping the head of the queue (the oldest) is then strictly better than dropping the tail (the newest). For a keyed buffer where only the latest value per key matters, dropping the oldest per key is better still: you reduce volume without losing any entity (a product, a user, a sensor).

4. Head-of-line blocking?

In a single FIFO (first in, first out) buffer, one slow item stops everything behind it. Giving each key or each tenant (customer) its own sub-buffer fixes this, at the cost of managing N buffers. This is the idea of bulkheading.

Bufferbloat: the thing most people get wrong

Bigger buffers feel safer. Past a certain size, they are not.

A large buffer that stays full gives you the latency of a full buffer all the time, and no extra throughput. Throughput was always μ. The buffer never changed it.

The correct way to think about sizing: a buffer should be sized for absorbing bursts, not for sustained overload. Use bandwidth-delay-product logic (the amount of data "in the pipe" during one round trip): the buffer must cover the length of the burst plus the reaction time of whatever downstream mechanism will actually fix the imbalance. If your buffer is chronically more than about half full, it is not protecting you. It is a queue on which you pay extra latency all the time. The right move is a smaller buffer plus an earlier signal, not a bigger buffer.

Two mitigations are worth knowing by name:

• CoDel (controlled delay). Measure the minimum queueing delay over a sliding time interval. If even the minimum stays above a target, the queue never drains: it is a standing queue, not a burst. Then start dropping. This is elegant because it separates a good burst-absorbing buffer from a bad standing one, which raw queue depth cannot do.
• Deadline-aware buffering. Stamp each record with an expiry time when it enters the buffer. When you take it out, drop it if it is already useless. This saves the consumer's scarce capacity for records somebody will still use. It is cheap to build and surprisingly effective.