Inference, run counts and the two variances
Why is the run the sample rather than the operation?
Inference as the object of study
You now have correct observations. This article asks whether the difference between two sets of correct observations is real. The object of study is no longer the system. It is your own inference process. The failure is no longer a mismeasured system. It is a confident conclusion drawn from noise.
The governing constraint is unforgiving: you have a finite number of runs and an unbounded number of ways to interpret them, and the interpretation is chosen by a person who wants a particular answer. Every technique below is a device for removing degrees of freedom (free choices) from that person.
Noise is not the enemy. Noise you have not quantified is, because an unquantified noise floor lets any difference look like a discovery.
Run counts and the two variances
There are two different sources of variation. They have different sizes, and mixing them up is the most common error here.
Within-run variance is the spread of individual operations inside one execution: the latency distribution from The right quantity. It is large, well sampled, and mostly a property of the system.
Run-to-run variance is the spread of a summary statistic (for example the median or the p99) across repeated executions of the identical configuration. It comes from where the instance was placed, thermal state, cache warmth, neighbours on shared hardware, JIT (just-in-time compiler) decisions, page-cache leftovers, and whatever else the machine remembers. It is usually poorly sampled (three runs is typical), and it is the variance that decides whether an A/B difference is real.
The error is to use within-run variance to claim confidence in a between-run comparison. Millions of operations in one run give a very tight confidence interval on that run's median. It is tempting to read this tightness as precision about the configuration. It is not. It is precision about that execution. The run is one sample from the configuration's distribution, and one sample (n = 1) has no error bar at all. Independence is assumed but false: the operations inside a run are not independent samples of the configuration, because they all share one realization of every environmental variable.
This gives a budget rule. If run-to-run variance dominates (it usually does on cloud instances), then more operations per run buys almost nothing, and more runs buys everything. Five short runs beat one long run for comparison. One long run is better for describing the shape of the distribution. Different questions need different allocations.