Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Concurrency limits and slots

How do you find a concurrency limit when capacity is unknown?

Slots are static admission: divide capacity by an assumed per-task footprint, and count. Simple, and wrong in exactly the way fixed granularity is always wrong. A task that needs 0.3 of a slot wastes 0.7. A task that needs 1.4 either fails or steals from its neighbor. That is the granularity mismatch again, now in the control plane, and it is the reason slot-based systems drift toward resource vectors (CPU, memory, ... per task).

Adaptive limits: AIMD

The adaptive alternative is borrowed wholesale from networking. Measure latency. Additively increase the while latency is flat. Multiplicatively decrease it when latency rises. This is AIMD (additive increase, multiplicative decrease), the rule used by TCP congestion control, applied to admission instead of packets. Gradient limiters are literally derived from TCP Vegas, a delay-based TCP variant.

# Run once per interval, using measured latency of the last interval.
if latency <= baseline_latency * 1.2:     # latency is "flat"
    limit = limit + 1                      # additive increase
else:                                      # latency is bending upward
    limit = max(1, int(limit * 0.9))       # multiplicative decrease

The reason it works here as it does in networks: the only locally measurable signal for a global capacity limit is your own latency getting worse, so every system that needs to find an unknown ceiling ends up probing it the same way.

When capacity is unknown, you find it by increasing until latency bends, and