Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Reproducibility versus validity; pinning environments

Why are repeating a result and generalizing it different questions, and how far does a container pin?

Two different questions

Everything before this article looked inward: was the measurement taken correctly? This article looks outward: what does the measurement allow you to believe about situations it never touched?

Reproducibility and validity are often bundled together, and in a useful sense they are opposites:

  • Reproducibility asks whether a result survives repetition (the same experiment, run again).
  • Validity asks whether a result survives extrapolation (a different situation from the one measured).

A perfectly reproducible result can be entirely invalid: a stable, repeatable measurement of the wrong thing.

The governing constraint is informational and goes in one direction. A measurement is evidence about the conditions it was taken under and about nothing else. Every sentence you write beyond those conditions is an argument. Arguments can be good. But they are a different kind of object from data, and the typical failure of this whole mechanism is arguments laundered as measurements.

The experiment produces a fact about one configuration; everything you want from it is an argument you still have to make.

Pinned versions, fixed seeds, containerized environments

These are the mechanics of repeatability. They follow from one observation: a result is a function of its inputs, and any input that is not pinned is a free variable that will vary.

There are three classes of free variable, and each fails in a different way.

  1. Versions drift silently. An unpinned dependency resolves to something different tomorrow. Default values change between minor releases. So "the same engine" is not always the same engine.
  2. Randomness varies from run to run: data generation, , hash seeds, the order of a shuffle. If the random seed (the starting value of a random number generator) is not fixed, two runs differ in workload as well as in system state.
  3. Environment (kernel, system libraries, CPU flags, filesystem) is usually invisible until it is the whole explanation.

Containers (isolated environments such as Docker) pin the second and third classes, but people trust them too much. A container does not pin the kernel, because containers share the host's kernel. It does not pin the CPU generation, the microcode or the available instruction sets. And it adds a filesystem layer and a network layer whose overhead depends on the workload and is not zero. A container is a strong control over user-space software and no control at all over the machine.

The seed question has a subtlety. Fixing the seed makes a run repeatable. But it makes the result a single realization: you removed workload from the comparison. That is right for an A/B contrast, where you want both sides to see the same data. It is wrong for describing how the system behaves in general, because it hides sensitivity to the data. The correct pattern has two steps:

  • Use a fixed seed per configuration pair for the comparison.
  • Then run a sweep over several seeds to check that the conclusion is not an artifact of one generated dataset.

The principle "selection consumes evidence" applies here too: if you tried several seeds and reported the best one, the seed became a selection.