The harness is not neutral: open versus closed loop
Why does a closed-loop load test let the system choose its own arrival rate?
Why the harness is no longer neutral
Experiment design designed the contrast. This article builds the machine that applies it. In experiment design, error entered through what you failed to pin down. Here it enters through a system you built yourself, which has queues, concurrency limits, a scheduler and a clock. It therefore inherits every problem that real systems have.
The new constraint is closed-loop coupling: if the generator waits for the system, the system controls its own input, and the experiment measures a joint system that does not exist in production.
What stops mattering is realism of content, which Experiment design already settled. What matters now is realism of arrival: when and how fast requests come.
The system under test cannot be allowed a vote on how much load it receives, because in production it does not get one.
Open-loop vs. closed-loop
This is Little's Law used as a design decision and not as an analysis tool. Recall L = λW: L is the number of requests in the system, λ is the arrival rate, W is the time each spends inside. A load model is a choice about which one of the three you hold fixed.
Closed-loop fixes L. There are N virtual users. Each sends a request, waits for the response, then sends the next. Little's Law then forces λ = N / W. When the system slows down, W rises and λ falls automatically. The generator slows itself exactly in sympathy with the system. That is why a closed-loop test can never push a system into overload: it is a negative feedback loop (a loop that cancels change) with the system under test inside it.
Open-loop fixes λ. Requests are generated on a schedule that does not depend on any response. L is then free to grow without limit, which is what happens when a queue backs up in production.
Same example, open-loop. At λ = 5,000 requests/s and W = 400 ms, L = 5,000 × 0.4 = 2,000 requests in the system at once, and still rising if the system cannot keep up.
Which model is right depends on whether your real clients are a finite population that blocks (waits for each answer). They almost never are. Web users, upstream services with their own retry logic, and Kafka producers writing at whatever rate events occur: none of them wait for you before generating the next piece of work.
The reality is usually partly-open: arrivals are independent, but individual users give up after a timeout, which limits L in a way neither pure model describes.
The practical consequence is not subtle. Closed-loop testing systematically underestimates queueing and hides the knee entirely, because it cannot reach it. A system that looks stable at "200 concurrent users" may collapse at the equivalent arrival rate, and you will not find out until production.
This is the first time Little's Law dictates the shape of an instrument and not only the explanation of a result.
The one legitimate use of closed-loop: when your real client truly is a fixed pool that blocks, such as a connection-pooled batch job or a fixed set of worker threads. Then the closed-loop model is the production model.