Queues explode near full load
Response time explodes as utilization nears 100%.
What it is
For one server with random arrivals and random service times, the average response time is
where S is the average service time. Do not memorize the formula; look at the shape:
| Utilization ρ | Response time R |
|---|---|
| 0.5 (50%) | 2 × S |
| 0.8 (80%) | 5 × S |
| 0.9 (90%) | 10 × S |
| 0.95 | 20 × S |
| 0.99 (99%) | 100 × S |
It is a hyperbola with a wall at 100%, so **the last few percent of utilization cost more than the first eighty.
From Physical Substrate & Performance Laws › Little's Law and queues
The headroom argument is the practical point. At ρ = 0.8, a 25% traffic increase gives ρ = 1.0: the queue grows without limit. At ρ = 0.5, the same 25% gives 0.625 and response time rises only from 2 × S to 1/(1 − 0.625) = 2.67 × S. You do not buy average speed with headroom. You buy the ability to absorb variance without falling off a cliff.
From Physical Substrate & Performance Laws › Little's Law and queues
Queueing law. Contention-allocated resources cost 1/(1 − ρ). Anything whose capacity is shared and claimed opportunistically has a cost curve that is flat, then nearly vertical. Design the operating point, not the capacity.
From Core Data Structures › Hash tables
Where it appears
In 8 mechanisms. Each says what the mechanism is about, then how this principle shows up there, in its own words.
- Data Movement
How do records cross between two systems that have independent rates, independent availability and independent lifetimes, and what are you willing to lose when the assumption underneath the crossing breaks?
Goodput, not throughput, is the objective.
- Core Data Structures
How do we choose a data structure when the real cost is set by the hardware, not by Big-O?
Contention-allocated resources cost 1/(1 − ρ).
Read it inHash tables
- Transactions & Concurrency Control
What happens when many users change the same data at the same time, and how do we keep the result correct?
The metric you need is goodput, not throughput.
Read it inOptimistic concurrency control (OCC)A frame for placing any mechanism
- Memory & Resource Management
Who is allowed to hold how much memory, for how long, and what happens to the work when the answer is less than it asked for?
The degradation path is where the system collapses, because it is the one resource nobody gated.
Read it inGraceful degradation, the honest versionGC tuning versus manual management
- Fault Tolerance & Delivery Semantics
What is the cheapest thing that can be safely done twice?
The consequence is important: the utilization target of a system that recovers by replay is set by its recovery budget, not by its latency budget.
- Emergent Failure Dynamics
What happens when nothing is broken, every component is healthy, and the system's own response to stress makes the stress worse?
Recovery time is the outage times ρ/(1−ρ), and catching up at full speed is an overload event aimed at everything downstream.
Read it inBacklog burst and replay stormsThe loop as an objectWhat feedback loops forbid
- Scheduling & Workload Management
Which work runs now, which work waits, and who pays for the waiting?
> EDF is optimal exactly up to utilization 1 and catastrophic immediately after.
Read it inDeadline-aware schedulingThe gate
- Measurement & Experimental Method
What would have to be true for this number to mean what I think it means?
List the resources first; utilization without saturation is not a problem, and saturation without utilization is the interesting case.
Read it inThe USE method and per-operator profilesSaturation, backlog and recovery
Related
Made on the same pages most often.