Pinning, isolation and cold versus warm

Which versions, machine state and warm-up point must be fixed or reported for a result to repeat?

Version and config pinning

The configuration space is so large that the default values are the experiment. Defaults also change silently between minor versions. Two runs a month apart on "the same engine" may differ in a default you never knew existed. Your A/B comparison then contains an invisible time axis, and the contrast is not the one you designed.

The key point: hand-written configuration notes record intentions. A machine-generated dump records reality. The two differ exactly where the interesting bugs are. So the dump must be mechanical and complete: engine and runtime versions, kernel, instance type and generation, storage class, and every non-default setting. Capture it at run time and attach it to the result, not to the write-up.

A small example of a mechanical dump:

# Run at the start of every experiment; store next to the result files.
{
  date -u +%FT%TZ
  uname -a
  lscpu | head -20
  cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null
  curl -s http://169.254.169.254/latest/meta-data/instance-type 2>/dev/null
  tessera --version
  tessera config dump --all          # every setting, defaults included
  git rev-parse HEAD                 # exact benchmark code version
} > "results/$RUN_ID/environment.txt"

This connects to the principle that whoever chooses the conditions chooses the result: mechanical capture is what lets a reader audit the conditions.

Environment isolation

This is a physical problem with an economic twist. Any environment whose performance depends on its own recent history turns run order into an independent variable you did not declare.

Credit-based cloud instances are the clearest case. The credit balance (CPU or I/O allowance earned while idle and spent while busy) is a hidden state variable. The same workload gives two different answers depending on what ran in the previous hour.
Thermal throttling (the chip slows down when hot) does the same, more slowly.
Turbo behaviour (a chip runs faster when few cores are active) makes a single-core measurement not comparable to an all-core one.
Neighbour tenants add a variable that is not only unmeasured but also unobservable to you.

There are two escapes, and they are not equivalent.

Eliminate the memory. Use dedicated hardware, pinned clock speeds, turbo disabled. This is expensive and often unavailable.
Randomize and interleave run order, so that history becomes noise instead of bias. This costs variance, and therefore more runs.

When neither is possible, measure the environment instead of assuming it. Interleave a fixed, known baseline workload with your experimental runs and report its run-to-run spread as the noise floor. If the effect you are chasing is smaller than the drift of that floor, you do not have a result, and no analysis will produce one. This is the "instrument is a system" principle: the environment is part of the instrument.

Cold vs. warm start as an explicit condition

Nearly every system installs something that fills up over time: the page cache, the buffer pool (the database's own cache), JIT-compiled code, connection pools, metadata caches, warm pools of ready machines. So performance is not a number but a trajectory, and the average over that trajectory describes no situation that ever occurs. Cold and warm can differ by one to two orders of magnitude when the difference is a remote object store versus a local cache (for example, 100 ms versus 1 ms).

Which condition is correct is decided by the deployment question, not by convenience:

Ad-hoc analytics on scale-to-zero compute (machines that shut down when idle) is genuinely cold most of the time.
A long-running service is genuinely warm.
An autoscaled system is both, plus the transition. The transition is where the congestion pathologies of overloaded systems live, so it is the interesting case precisely because nobody measures it.

"Warm" is also not a yes/no state. Warm for one query shape can be cold for another, because what is resident in cache is specific to the queries that ran. The procedure is: warm up, discard, detect steady state with a stabilizing moving statistic, then measure. Report the discard rule as part of the result, because it is one of the thousand degrees of freedom described earlier in this article.