Coordinated omission and steady state

How does a load generator silently drop the slow requests, and how do you detect steady state?

Coordinated omission

This is the most important measurement bug in the field. It is head-of-line blocking happening inside the ruler.

The mechanism. A generator intends to send 1,000 requests per second, one every 1 ms. The system stalls for 1 second. A naive generator has one request outstanding at a time. It blocks for the full second, then resumes. It records one sample of 1,000 ms. The roughly 1,000 requests that should have been sent during the stall were never sent, so they contribute nothing. The recorded data contains one bad sample where the truth contains a thousand. The p99 is computed over a sample set from which the worst observations have been deleted by the very event that made them bad.

The distortion is multiplicative and grows with stall length, so it is worst exactly where you care most. Reported p99 values can be off by one to three orders of magnitude. It is the single largest source of wrong latency numbers in published results, and it is usually accidental, not dishonest.

Two corrections. They are not equivalent.

1. Prevention. Issue requests on the schedule, regardless of whether earlier responses have come back. Measure latency from the intended send time, not the actual send time. The difference is the queueing delay the client experienced, and it is real latency.
2. Post-hoc compensation. Synthesize the missing samples by back-filling the expected arrivals during the stall. This is a reconstruction that depends on an assumed arrival model, so it is strictly weaker than measuring correctly. Prefer the first. Use the second only for data you already have.

A small sketch of prevention in Python (the idea, not a full tool):

import asyncio, time

async def open_loop(rate_per_s, duration_s, send):
    interval = 1.0 / rate_per_s
    start = time.perf_counter()
    latencies = []
    async def one(intended):
        await send()                                  # the real request
        latencies.append(time.perf_counter() - intended)   # from INTENDED time
    tasks = []
    n = int(rate_per_s * duration_s)
    for i in range(n):
        intended = start + i * interval
        delay = intended - time.perf_counter()
        if delay > 0:
            await asyncio.sleep(delay)
        tasks.append(asyncio.create_task(one(intended)))  # do not wait for reply
    await asyncio.gather(*tasks)
    return latencies

The general rule. Any measurement taken by a component that shares fate with the thing it measures is censored (cut off) in the optimistic direction. Examples: client-side metrics from a failing client; application metrics from a process that is stalled by garbage collection; a monitoring agent starved by the same CPU pressure it should report. This sharpens the principle "the instrument is a system": it is often the same system.

Think time, ramp-up, warm-up discard, and steady-state detection

Four separate mechanisms are usually collapsed into the phrase "let it settle." They should be kept apart.

Think time is the pause a simulated user takes between requests. It exists to stop a closed-loop generator from being a pathological arrival process. Constant think time produces synchronized clients: all users act at the same instant. This is the thundering-herd pattern (many clients hitting a service at once), manufactured in your own harness. Exponentially distributed think time approximates independent arrivals. It is the same reason jitter (a random delay) is added to retry backoff. Independence does not arise by itself. It has to be constructed.

Ramp-up serves two purposes that are worth separating. First, it fills the caches and pools described in Experiment design and avoids a cold-start artifact at time zero that would contaminate the run. Second, and less obvious, ramp-up is a measurement in its own right. The transition is where autoscaling lag, connection storms and cold-cache collapse show up. Discarding it entirely throws away the most interesting failure mode.

Steady state must be detected, not assumed. The detection rule is one of the degrees of freedom from Experiment design, so fix it in advance. A workable rule: a moving statistic (throughput, or better, a mid-percentile such as the median) stops trending over a window that is a multiple of the system's slowest time constant. The trap: some systems never reach steady state under the applied load, because the load exceeds capacity. The moving average keeps drifting. Reading the drift as "needs more warm-up" is a classic misreading of a saturated system.