Beyond staging: production, cost and explanation
What can only an experiment on the real system show, and what turns a measured difference into a result you can reuse?
Experimenting in production
Every form of staging is a model of production, and models leave out exactly what nobody thought to include: the real key distribution, the real mix of concurrent requests, the real retry behaviour of clients, the real neighbouring workloads, the real data volume. So the staging environment is systematically the optimistic case. You cannot fix this by building a better staging environment. You can only reduce the gap by moving the experiment closer to the real system.
The mechanisms differ in how much reality they buy and what they risk:
| Mechanism | What it is | What it buys | What it cannot do or risks |
|---|---|---|---|
| Shadow traffic | Duplicate real requests to a candidate system and discard its responses | The real arrival process and real data, with zero user risk | Cannot measure anything the discarded response would have caused: no write path to real state, and no feedback from clients that react to latency |
| Canarying | Send a small fraction of real users to the candidate | The full loop on a small population | Exposure: users can be hurt, but the harm is bounded by the fraction |
| Interleaved live A/B | Randomly assign each request to arm A or arm B at the same time | The strongest statistical design available: paired, randomized, same hardware generation, same moment in time, which removes every environmental confounder that Experiment design fought | Full exposure of those users, and the treatment must be assignable per request |
The threat that belongs to this family is interference between arms. The two arms share caches, share a database and share a connection pool. A heavy treatment arm slows down the control arm, which biases the comparison toward the treatment. The clean-contrast assumption of Experiment design fails here not by accident but by construction. The standard remedy is to isolate the arms, and that brings back exactly the hardware confound you came here to escape.
Cost as a reported dimension
Standard benchmarks treated the omission of cost as a benchmarketing sin. It is more than that. Cost is an axis in a different unit (money, not time), and it makes every other result conditional. A 2× speedup on 3× the hardware is a regression. Two systems are comparable only at equal cost or at equal performance, never at equal hardware, because different architectures reach the same result through different mixes of resources.
The reporting discipline that follows is to normalize to a unit of value: dollars per query, per TB scanned, or per thousand events. Then report the frontier and not a single point. This also makes the result perishable in a way that pure performance numbers are not, because prices change independently of engineering. That is why almost nobody reports it, and why the results that do report it are worth more.
Mechanistic explanation as the acceptance test
This is the discipline that separates a finding from a coincidence, and it is the one most often skipped, even in otherwise careful work. A measured difference with no causal account is an unexplained correlation, and unexplained correlations in systems are often artifacts: a cache state, an accident of data placement, a version mismatch.
The test is predictive, not a story. Your explanation implies that something else must also be true: a specific counter should move, a specific configuration should remove the effect, a specific workload change should make it larger. Check that second prediction. If it holds, you have a mechanism. If it does not, you have a story that happened to fit one dataset.
This has the same structure as pre-registration (Experiment design) and as the held-out confirmation run (Statistical hygiene): the prediction is made before looking at the new evidence, so the evidence cannot be bent to fit. It is also what makes a result transferable, because you can only reason about a new situation through the mechanism, never through the number.
Modelling as a partner of measurement
Queueing models, the roofline model and the Universal Scalability Law are cheap experiments that need no hardware. They are strictly weaker as evidence and strictly stronger as explanation. A model tells you why a curve has its shape and applies across the whole parameter space. A measurement tells you one point with certainty.
The productive relation is not "either/or":
- Use the model to predict.
- Measure two or three points to validate it.
- Use the validated model to interpolate the rest.
That turns a scale ladder from a dozen runs into three. This is the "fixed cost sets batch size" rule applied once more to experimental effort.
Negative results and the internal corpus
"Whoever chooses the conditions chooses the result" was framed around vendors. But the same filter works inside an organization. An engineer who spends a month proving that an approach does not work has produced valuable information and no artifact that anyone will read. The finding is not recorded, and the next engineer repeats the month.
So the collection of internal measurement knowledge is filtered by survivorship in exactly the same way as the public one, by the same mechanism, and without any of the commercial motive that people blame for it. This is the organizational form of "selection consumes evidence". It is why a measurement culture is a filing problem before it is a methodology problem.