Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

How flow-control mechanisms compose

How do these mechanisms stack by position and timescale, and which combinations cause trouble?

Composition: how these fit together

A production pipeline uses most of these at the same time, at different layers and different timescales.

By position:

external client
  ↓  rate limit + admission control     (boundary you don't control)
edge / gateway
  ↓  deadline propagation, priority tagging, early shedding
service
  ↓  bulkheaded pools + circuit breaker per dependency
  ↓  bounded buffer, adaptive batching
durable buffer  <-- backpressure TERMINATES here (deliberate absorption point)
  ↓  pull-based consumption (structural backpressure)
processing
  ↓  credit-based flow control
sink

By timescale (the worth keeping):

HorizonMechanism
microseconds to millisecondsbuffering,
milliseconds to seconds, credit windows
secondsshedding,
seconds to minutescircuit breaking
always onrate limits,

They are not alternatives. They are defense in depth: each layer handles what the faster one could not.

Interactions that cause trouble

  • Retrying a batch retries the good records too. A at layer 3 triggers a at layers 2 and 1, and one logical request becomes 8 attempts (2 × 2 × 2). Retry at one layer only, and set each outer timeout strictly larger than the inner timeout times the number of attempts.
RelationshipRetry amplification across layers →
attempts each layer makes for one call
layers that retry
attempts one logical request becomes
each outer timeout strictly larger than the inner timeout times the number of attempts
2 attempts per layer3 attempts per layer
Data table
Attempts one logical request becomes, by retrying layers
Layers that retry LseriesAttempts
12 attempts per layer2
1.062 attempts per layer2.09
1.132 attempts per layer2.18
1.192 attempts per layer2.28
1.252 attempts per layer2.38
1.312 attempts per layer2.48
1.382 attempts per layer2.59
1.442 attempts per layer2.71
1.52 attempts per layer2.83
1.562 attempts per layer2.95
1.632 attempts per layer3.08
1.692 attempts per layer3.22
1.752 attempts per layer3.36
1.812 attempts per layer3.51
1.882 attempts per layer3.67
1.942 attempts per layer3.83
22 attempts per layer4
2.062 attempts per layer4.18
2.132 attempts per layer4.36
2.192 attempts per layer4.56
2.252 attempts per layer4.76
2.312 attempts per layer4.97
2.382 attempts per layer5.19
2.442 attempts per layer5.42
2.52 attempts per layer5.66
2.562 attempts per layer5.91
2.632 attempts per layer6.17
2.692 attempts per layer6.44
2.752 attempts per layer6.73
2.812 attempts per layer7.03
2.882 attempts per layer7.34
2.942 attempts per layer7.66
32 attempts per layer8
3.062 attempts per layer8.35
3.132 attempts per layer8.72
3.192 attempts per layer9.11
3.252 attempts per layer9.51
3.312 attempts per layer9.93
3.382 attempts per layer10.4
3.442 attempts per layer10.8
3.52 attempts per layer11.3
3.562 attempts per layer11.8
3.632 attempts per layer12.3
3.692 attempts per layer12.9
3.752 attempts per layer13.5
3.812 attempts per layer14.1
3.882 attempts per layer14.7
3.942 attempts per layer15.3
42 attempts per layer16
4.062 attempts per layer16.7
4.132 attempts per layer17.4
4.192 attempts per layer18.2
4.252 attempts per layer19
4.312 attempts per layer19.9
4.382 attempts per layer20.7
4.442 attempts per layer21.7
4.52 attempts per layer22.6
4.562 attempts per layer23.6
4.632 attempts per layer24.7
4.692 attempts per layer25.8
4.752 attempts per layer26.9
4.812 attempts per layer28.1
4.882 attempts per layer29.3
4.942 attempts per layer30.6
52 attempts per layer32
5.062 attempts per layer33.4
5.132 attempts per layer34.9
5.192 attempts per layer36.4
5.252 attempts per layer38.1
5.312 attempts per layer39.7
5.382 attempts per layer41.5
5.442 attempts per layer43.3
5.52 attempts per layer45.3
5.562 attempts per layer47.3
5.632 attempts per layer49.4
5.692 attempts per layer51.5
5.752 attempts per layer53.8
5.812 attempts per layer56.2
5.882 attempts per layer58.7
5.942 attempts per layer61.3
62 attempts per layer64
13 attempts per layer3
1.063 attempts per layer3.21
1.133 attempts per layer3.44
1.193 attempts per layer3.69
1.253 attempts per layer3.95
1.313 attempts per layer4.23
1.383 attempts per layer4.53
1.443 attempts per layer4.85
1.53 attempts per layer5.2
1.563 attempts per layer5.57
1.633 attempts per layer5.96
1.693 attempts per layer6.38
1.753 attempts per layer6.84
1.813 attempts per layer7.32
1.883 attempts per layer7.85
1.943 attempts per layer8.4
23 attempts per layer9
2.063 attempts per layer9.64
2.133 attempts per layer10.3
2.193 attempts per layer11.1
2.253 attempts per layer11.8
2.313 attempts per layer12.7
2.383 attempts per layer13.6
2.443 attempts per layer14.6
2.53 attempts per layer15.6
2.563 attempts per layer16.7
2.633 attempts per layer17.9
2.693 attempts per layer19.2
2.753 attempts per layer20.5
2.813 attempts per layer22
2.883 attempts per layer23.5
2.943 attempts per layer25.2
33 attempts per layer27
3.063 attempts per layer28.9
3.133 attempts per layer31
3.193 attempts per layer33.2
3.253 attempts per layer35.5
3.313 attempts per layer38.1
3.383 attempts per layer40.8
3.443 attempts per layer43.7
3.53 attempts per layer46.8
3.563 attempts per layer50.1
3.633 attempts per layer53.6
3.693 attempts per layer57.5
3.753 attempts per layer61.5
3.813 attempts per layer65.9
3.883 attempts per layer70.6
3.943 attempts per layer75.6
43 attempts per layer81
4.063 attempts per layer86.8
4.133 attempts per layer92.9
4.193 attempts per layer99.5
4.253 attempts per layer107
4.313 attempts per layer114
4.383 attempts per layer122
4.443 attempts per layer131
4.53 attempts per layer140
4.563 attempts per layer150
4.633 attempts per layer161
4.693 attempts per layer172
4.753 attempts per layer185
4.813 attempts per layer198
4.883 attempts per layer212
4.943 attempts per layer227
53 attempts per layer243
5.063 attempts per layer260
5.133 attempts per layer279
5.193 attempts per layer299
5.253 attempts per layer320
5.313 attempts per layer343
5.383 attempts per layer367
5.443 attempts per layer393
5.53 attempts per layer421
5.563 attempts per layer451
5.633 attempts per layer483
5.693 attempts per layer517
5.753 attempts per layer554
5.813 attempts per layer593
5.883 attempts per layer635
5.943 attempts per layer681
63 attempts per layer729
Stated in 5 places →Principle: Local optimizations compose badly →
  • Buffering hides backpressure. A large buffer between two stages means the downstream's trouble does not reach the upstream until the buffer is full. By then you are in serious trouble with no time to react. If you want backpressure to work, keep buffers small. If you want absorption, accept that you have delayed the signal, and watch buffer depth as your early warning.
  • Rate limiting upstream of a queue you also backpressure gives two independent limiters that swing against each other. Pick one authority per link.
  • Circuit breaker plus retry. The breaker must count retries as evidence. Otherwise you retry against an open path and never trip it.
  • Shedding without deadline propagation means you cannot shed intelligently, only randomly. Pass deadlines from the edge. It is the enabler for the best version of half the mechanisms in this article.

What to carry out of this article

Three things.

Buffers convert rate problems into time problems and never solve them. The is the design.

Backpressure is transitive and does not respect your boundaries. Know where it ends, and put a deliberate there.

Past saturation, goodput falls as load rises. Shedding and admission control are not concessions. They are what keeps the curve flat. A system with no deliberate rejection path has an involuntary one, and it is worse.