Saturation, backlog and recovery

How do you find where a system saturates, and what should you measure past it?

Saturation discovery

For a queue, response time grows without limit as utilization ρ approaches 1. The knee is the utilization at which the curve turns sharply upward. Where the knee sits depends on service-time variance (how uneven the processing times are), and variance cannot be looked up. So the knee must be found empirically. Assuming "80%" because a textbook says so is folklore applied without its condition.

Procedure. Sweep the arrival rate upward in an open-loop model, and record throughput, the latency distribution and the error rate together. Three regimes appear:

Regime	Throughput	Latency	Notes
Below the knee	Follows the offered load	Flat	The system keeps up
At the knee	Still follows	Climbs steeply	The only operationally useful number is here
Past the knee	Plateaus or falls	Explodes, errors rise	Congestion collapse; coordination costs (the USL β term) dominate

Step size matters more than people expect. Coarse steps jump over the knee and produce a curve with two regimes instead of three, which loses the single most useful number in the experiment. Each step must also run long enough to reach steady state at that rate, which makes the sweep expensive. Resolution is bought with time.

One caution: sweep upward and also downward. Some systems are metastable: once pushed into a bad state (for example, a retry storm), they stay there even after the load drops. They show hysteresis: the rate at which they break is higher than the rate at which they recover. So the up-curve and the down-curve differ. That gap is the metastability, and a one-direction sweep cannot see it.

Backlog vs. arrival rate

What do you do when the system cannot keep up? That is the regime the previous section deliberately entered.

Past saturation, latency is no longer a property of the system. It is a function of how long you have been overloaded. The queue grows at (λ − μ), where μ ("mu") is the service rate (what the system can finish per second). So measured latency grows linearly with elapsed time, and the number you report is determined by the run duration.

A 10-minute run and a 60-minute run differ six-fold from an identical system. Reporting "p99 latency at 2× capacity" without the run duration is meaningless.

So in the overload regime, the meaningful quantities change identity:

Throughput becomes the drain rate μ, which is the real capacity number.
Latency is replaced by the backlog growth rate.
Error rate and shed rate become first-class, because a system with admission control (refusing some work to protect the rest) trades latency for rejection, and that trade is the whole design.
The thing worth measuring is recovery: after removing the excess load, how long does the system take to drain, and does it return at all? That is the direct empirical test for metastability.

Goodput (useful work, see The right quantity) also separates from throughput here. Under overload, retries mean that work done and useful work diverge, so throughput can stay flat while goodput falls to zero.

What this article forbids

Any latency percentile from a generator that blocks on responses without intended-time correction.
Any claim about overload behaviour from a closed-loop model.
A saturation point that was assumed and not swept.
A latency number from an overloaded system without its run duration.
Constant think time, which manufactures synchronization.
Discarding the ramp-up without noticing that you discarded the transition failure mode.

Escapes and their prices

Escape	What it buys	What it costs
Coordinated-omission compensation	Correction of data you already have	Dependence on an assumed arrival model
Partly-open model	Realism	A third parameter, the abandonment timeout, which must itself be justified
Fine-grained saturation sweep	The knee	Run time
Distributed load generator	Enough load to saturate a large system	A new coordination problem: the generators now have clock skew, stragglers, and their own aggregation step that can destroy percentiles

One principle

Case, resolvedConstructed case · Ingrid at Bellwether Tickets

p99 of 40 ms in the test, collapse at the on-sale

A 200-user load test reported p99 = 40 ms and stable, and the real on-sale collapsed with latencies in seconds. Where it started: Load generation →

"200 users, p99 = 40 ms, stable." Her test was closed-loop. When the service slowed, her 200 virtual users automatically sent fewer requests: with W rising from 40 ms to 400 ms, the arrival rate fell from 5,000 to 500 per second. The test could not reach the knee. She rebuilds it as an open-loop, partly-open generator with exponentially distributed think time and a 10-second abandonment timeout, and sweeps the arrival rate up in fine steps and then back down.

The one-second freeze that barely moved the p99. The old tool was blocking on each response, so it recorded one sample instead of about a thousand. It was coordinated omission. The new generator measures latency from the intended send time. The same stall now produces a p99 close to the true value.

On the sweep she finds the knee at a much lower arrival rate than the closed-loop test suggested, and on the way back down the service recovers only at a rate far below the one at which it broke. She has found the metastability that the on-sale hit.