Scenarios

Not a verdict. Pin and record versions, kernel, instance type and settings.

Check for burstable instances that run on CPU credits: interleave A and B runs, and add a baseline workload that detects when credits run out. State whether each run started cold or warm.

Run an A/A test to get the noise floor, and write the decision rule beforehand (for example, ship only if p99 during peak sales improves by at least 15% at equal cost).

From the confidence interval, not a single number.

Time: five short interleaved runs cost more than one long run, but give an interval you can use.

The test was closed-loop. With 200 users, rate = users ÷ response time, so when response time rose from 40 ms to 400 ms, the rate fell from 5,000 to 500 requests per second, and the test never reached the knee.

• Open-loop or partly open arrivals (independent of responses, with users abandoning after a timeout).
• Exponentially distributed think time.
• Latency measured from the intended send time.
• Detection of steady state.
• A sweep of the arrival rate up and back down, to find the knee and the rate at which the system recovers.

Metastable overload, where recovering needs a rate far below the one that caused the failure.

HDR histograms on each node: logarithmic buckets, with a ceiling above the largest timeout (for example 60 s).

Ship the histograms, merge them centrally, and compute percentiles last. Never store pre-aggregated means or per-node p99s. Count goodput at the client, against the original request.

A histogram with 3 significant digits from 1 µs to 60 s is a few tens of KB, so 500 nodes per minute is a few MB: cheap.

Sketches have bounded error; keep raw samples for experiments, where storing every value is affordable.

Probably a retry storm: server throughput counts duplicates while goodput collapses.

Completions at the client against the original request; the ratio of retries; and wall-clock minus CPU time (35% CPU suggests waiting, not computing). Then use off-CPU analysis to find the lock, queue or disk wait.

Mitigations belong elsewhere, but the measurement fix is to count "a customer got a ticket", not "the server completed a request".

Run an A/A test first: if identical configurations differ by about 8%, then 6% is inside the noise. Variance between runs, not within a run, governs the comparison.

The rule for dropping outliers must be declared before the runs, and outliers are often real behaviour (garbage collection, compaction). Report with and without them.

Interleaved and paired, with a fixed number of runs, and report the difference with a confidence interval.

Four times the runs only halves the interval, so decide with the stopping rule whether the answer is worth that.

With noise of about 5 points, the best of 50 draws sits about 2.2 standard deviations high: roughly 11 points from luck alone.

Run a fresh, held-out confirmation of the winner and report that number (here, something like 4%). Plan capacity with the confirmed number.

One extra run costs little next to a wrong capacity plan.

For the configuration dump; who tuned each system; which of the 99 queries were reported; cache state and run order; hardware and cost.

Which mechanisms TPC-DS stresses, against those your workload stresses.

Build your own benchmark from a trace of your workload, and compare at equal cost. A result without disclosure is not evidence.

Compute wall-clock minus CPU time. If most of it is waiting, take an off-CPU profile, and apply the USE method to every resource, including pools and locks.

The latency is outside the process (network, a downstream service, the client), which is itself a finding.

Take a differential profile against a version known to be good.

Versions and defaults, random seeds, kernel and CPU generation, instance type, cold or warm state, data scale, and whether the result was measured below a scale boundary.

A pinned configuration dump and a one-command script.

The scope the result holds in, and the next experiment.

1. Predict first, and register the decision rule.
2. Run offline with open-loop load.
3. Shadow production traffic, then a 1% canary with separate pools.
4. A node-kill test, with a hypothesis and a condition for aborting.
5. A cost curve from 1 to 30 times today's scale.

One page: the mechanism that explains the result, a second prediction it makes, the limits, and the next experiment.

Realism costs exposure to production risk.

Staging leaves out the real distribution of keys, retries, noisy neighbours and volume, so it is the optimistic case.

Move closer to production with shadow traffic and canaries, watching for interference between the groups, and add fault-injection tests for rare paths, because a recovery path nobody has tested should be assumed broken.