Regression to the mean and the stopping rule

Why does the best of many configurations disappoint, and when should you stop running?

Regression to the mean and the single lucky run

These two mechanisms are often named together, but they are different.

The lucky run is a selection effect. Take the best of five runs for A and the mean for B, and you have compared A's upper tail with B's centre. Reporting the best result is universal in casual benchmarking and almost universal in vendor material. It inflates the result by about the spread of the distribution, which is exactly the quantity you did not report.

Regression to the mean is stronger and less intuitive. Any configuration that was selected because it scored extremely will, when measured again, score closer to the average. Nothing changed. Its extreme score was partly luck, and luck does not repeat. This causes a specific, expensive pattern: a tuning sweep over 50 configurations picks the winner, the winner is deployed, and it performs worse in production than in the sweep. Nothing broke. The sweep's maximum was partly noise, and the wider the sweep, the more extreme the noise you select. The winner of a 50-way comparison is the configuration that was both good and lucky.

The defence is a held-out confirmation run. After selecting, measure the winner again, fresh, and report that number. This is the same logic as a held-out split in machine learning, and the same logic as pre-registration: keep the data used to choose separate from the data used to claim. It also gives the honest estimate of the improvement. That estimate will be smaller than the sweep suggested, and it is the number to plan with.

The stopping rule

This rule bounds everything above. Measurement has a cost. The decision it informs has a value. Rigour beyond what changes the decision is waste.

The rule compares two quantities. On one side is the cost of the next batch of runs in time and money. On the other is the expected value of the information, which is zero if no plausible outcome changes the decision. If a 15% improvement and a 25% improvement both lead you to ship, an experiment that separates them is worth nothing, however clean it would be.

Three practical stopping conditions:

The interval is tight enough that both ends lead to the same action. Stop: you have the answer.
The interval crosses the threshold of practical interest, and the noise floor makes narrowing it too expensive. Stop, and report that the question cannot be answered at this cost.
The effect is smaller than the noise floor. Stop immediately. More runs on a shared, drifting environment are runs thrown at a floor that will not move.

One hazard: a stopping rule that depends on the results, such as "run until significant", is optional stopping. It inflates false positives badly, because you keep sampling until noise happens to favour you. Fix the number of runs in advance, or use a sequential procedure designed for this. Otherwise this technique quietly undoes the pre-registration from Experiment design.

What this article forbids

Confidence claims about a configuration that rest on within-run variance.
Any A/B conclusion without a measured A/A floor.
Removing outliers after the fact when the conclusion depends on it.
Running all of A and then all of B.
Reporting the best run.
Reporting a tuning sweep's winner without a fresh confirmation run.
Running until the result becomes significant.
Treating statistical significance as practical significance.

Escapes and their prices

Escape	What it buys	What it costs
More runs	A narrower interval, but only at 1/√n: four times the runs halve the interval	Brute force gets expensive fast. This is why the noise floor matters more than the run count
Paired, interleaved design	Statistical power for free	The treatments must be runnable in alternation, which excludes anything with long warm-up or persistent state
Bootstrapping	Valid intervals on heavy-tailed data	You need enough runs to resample from
Dedicated hardware	A low noise floor	Direct cost: money substituting for other resources, here buying statistical power

One principle

Case, resolvedConstructed case · Ingrid at Bellwether Tickets

+14% in the sweep, +4% in production

The best of 50 Tessera configurations gave +14% in tests and +4% in production, and a 3-run A/B showing 6% tempted the team to drop a noisy run and keep running until it looked solid. Where it started: Statistical hygiene →

+14% in the sweep, +4% in production. The sweep's winner was selected for being extreme, so part of its score was luck. Ingrid now runs a held-out confirmation run of the winner on a fresh instance and reports that number, and she plans with it.

The 6% from 3 runs. She first runs an A/A test: the same configuration repeated gives a spread of 8%. A 6% difference sits inside the noise floor, so it is not a result yet. She re-runs in an interleaved, randomized, paired order (A, B, B, A, A, B, ...) with a run count fixed in advance, and reports the difference with an interval.

The dropped run. The outlier rule is now written down before the runs. The "noisy" run is explained, not removed (it hit a compaction stall, which will happen on sale day too), and both versions of the numbers go in the report.

"Run until significant." She fixes the number of runs first and applies the stopping rule: when the interval is tight enough that both ends lead to the same decision (ship only if the on-sale p99 improves by at least 15% at the same cost), she stops.