Backpressure propagation

Why can a stream engine slow its source safely, and how do you find the real bottleneck?

When an operator is slower than the one before it, with unbounded input and bounded memory there are exactly three options:

1. Drop records. This loses data.
2. Buffer without limit. This eventually runs out of memory.
3. Slow the source.

Only the third preserves correctness. So the engine uses bounded per-channel buffers plus credit-based flow control: a receiver tells the sender "I have room for k more records" (these are credits), and the sender stops when it has no credits. When one buffer is full, the operator in front of it cannot send, so its own input buffer fills, and so on, hop by hop, until the source stops polling the log.

The log absorbs it

This reveals the architectural point:

Without a replayable log upstream, backpressure degenerates into either data loss or an unbounded queue somewhere else. This is also where the recovery cost of this whole mechanism actually lands. The conserved-cost law says cost is relocated, and here it is relocated onto the source's retention bill, paid in storage dollars for however many hours of replay you want to be able to afford.

Three operational consequences

1. Topology throughput is the min over all operators, with no queueing rescue, because queues are bounded by design. The slowest operator sets the speed of the whole job.
2. Backpressure is transitive and therefore non-local. Every operator upstream of the bottleneck reports backpressure. So the culprit is the first operator that is not backpressured, reading downstream: its own output is slowed by the next operator, but it itself is not waiting.
3. Capacity planning must leave headroom. Arrival rate is variable, and a job running at 95% of capacity converts every burst into growing lag. Variance costs more than volume. This is the queueing wall (response time ∝ 1 ÷ (1 − ρ)) in a dataflow graph.