What kind of thing is a measurement?
Why is a measurement a fact about one configuration, and why does experiment design need its own method?
Every other mechanism on this platform is something that moves, stores, orders or protects bytes. This mechanism produces a belief. The output is not a byte but a justified claim: "this is faster", "this scales", "this costs less". A claim can be wrong in ways a byte cannot. That is the break from every other mechanism.
Two concerns that matter for ordinary systems stop mattering here:
- Data correctness. The data in an experiment is synthetic, disposable, and often deliberately unrealistic. The question is not "are the values right?" but "is the inference right?"
- Durability. Nothing that is measured needs to survive.
Three constraints of a different kind replace them.
1. The epistemic constraint (what you can know). The instrument is coupled to the thing it measures. A load generator is itself a distributed system with queues, connection pools, a scheduler and backpressure (the system pushing back when it is overloaded). So every problem that can happen in the system under test can also happen inside the ruler. Coordinated omission (Load generation) is not a bug in a stopwatch. It is head-of-line blocking (one slow item holding up everything behind it) happening in the client, and it silently deletes the observations you most cared about.
2. The statistical constraint. In ordinary engineering, variance (run-to-run spread) is a property of the system that you try to remove. Here variance is the main object of study, and it cannot be removed. Thermal state (how hot the chip is), cache warmth, JIT state, neighbour tenants (other customers on the same cloud machine) and leftover data in the operating system's page cache mean that repeated runs are not independent samples. Also, percentiles cannot be combined (you will see why in The right quantity). So you cannot average results, re-aggregate them, or rebuild a distribution from summaries that were taken before you knew your question.
3. The organizational constraint. Almost every published number was produced by someone who prefers one outcome. Whoever chooses the configuration chooses the result, and the reader can audit only what was disclosed. This is the first topic where incentive is part of the mechanism and not a footnote.
The governing question of this mechanism is: what would have to be true for this number to mean what I think it means?
A correction before we start
"All benchmarks lie" is folklore, and it is lazy. Benchmarks rarely lie about what they measured. They are precise measurements of a narrow configuration. The usual failure is generalization: the reader extends a valid result to a workload, a scale or a hardware generation that the experiment never touched.
Treating benchmarks as dishonest lets you ignore them. Treating them as scope-limited forces you to ask what the scope was, and that is the useful question.
A related point: "p99 latency" is not a well-formed quantity unless you also state an aggregation window (over what time span?) and a composition rule (how are results from several machines or periods combined?). Most reported p99 numbers have neither.
Three principles that shape everything
Why experiment design needs its own method
For a mechanism you can reason from a constraint (a disk cannot rotate faster than its motor). You cannot do that here, because the thing you are designing is not a system but an inference, and inferences fail for reasons that have no physical analogue.
The central fact: the number of possible confounders is unlimited, but the number of things you thought to hold fixed is finite. The real job of the design is to shrink that gap before the run, not to argue about it afterwards.
An experiment is a machine for making one explanation survive and the others die. If no outcome would have killed your explanation, you built a demonstration instead.