Noise floor, effect size, outliers and run order

Why measure identical configurations first, declare outlier rules in advance and interleave runs?

Confidence intervals, effect size, and the noise floor

The order of steps matters, and in practice it is usually reversed.

First, establish the noise floor. Run the same configuration repeatedly and watch the spread of the summary statistic. This is the A/A test. It is the highest-value habit in this article, because it is the only way to learn what a "no difference" result looks like on your hardware. If A versus A shows 8% spread, then a 5% A versus B improvement is not a result, however clean the numbers look.

Only then does effect size mean anything. Report the difference together with its interval, for example "12% faster, 95% CI [7%, 17%]". (A 95% confidence interval is a range built so that, if you repeated the whole procedure many times, about 95% of such ranges would contain the true value.) A point estimate alone gives the reader no way to tell a solid finding from a coin flip.

Two corrections:

Statistical significance is not practical significance. With enough runs, a 0.4% difference becomes statistically significant, and a 0.4% difference is not worth a rewrite. State the threshold of practical interest in advance (this is the decision rule from Experiment design) and treat significance as necessary but not sufficient.
The standard interval formulas assume roughly independent, roughly well-behaved samples. Systems measurements break both assumptions (heavy tails, correlated runs). So the usual parametric intervals on tail statistics are unreliable. Bootstrap resampling over runs is the practical escape: draw many new sets of runs, with replacement, from the runs you have, and look at the spread of the statistic. Its limit is that it cannot create information that three runs do not contain.

Outlier policy declared in advance

The mechanism is degrees of freedom. It is the same one as pre-registration in Experiment design, applied to one especially tempting choice.

Removing outliers after seeing the data cannot be tested: you can always find a plausible reason to discard the run that disagrees with you. "The instance was clearly noisy." Maybe it was. But the decision was made after seeing which direction it pushes the conclusion, and that is the whole problem.

A deeper issue is specific to systems and matters more than the procedural one: outliers are often the signal. One 2-second sample among 5 ms responses may be GC, a leader election, a compaction stall (background rewriting of storage files), a metadata refresh, or a cold cache. These mechanisms exist, will recur in production, and are exactly what tail-latency work targets. Discarding them gives a measurement of the system's best behaviour, labelled as the system's behaviour. Standard statistical outlier tests assume one single generating process. Multi-modal systems break that assumption by construction.

The workable policy:

Declare the rule before the run.
Prefer to explain an outlier rather than remove it.
When you do remove one, report the results both with and without it.
If the conclusion depends on the removal, the conclusion is about the removal.

Randomized and interleaved run order

Experiment design showed that environments have memory, so the order of runs is a hidden variable. This is the remedy, and its logic is precise.

The failure it prevents: if you run all of A and then all of B, any steady drift in the environment (heat, neighbour load, credit depletion on burstable instances, background maintenance) is perfectly mixed up with the treatment. The data cannot tell drift from treatment, and no statistics can separate them, because the information was destroyed when the data was collected.

Interleaving (A, B, A, B, ...) turns drift from a bias into a shared offset that mostly cancels in the paired difference.
Randomizing the order stops any periodic environmental effect from lining up with a regular alternation.
Pairing lets you analyze differences instead of levels. This removes the variation between pairs from the comparison, so a paired test on interleaved runs can detect effects that an unpaired test with the same number of runs cannot.

This is the cheapest technique in this article. It costs only scheduling discipline, and it is the one most often skipped, because scripts naturally run things one after another.