Summary and key takeaways
The one question. What would have to be true for this number to mean what I think it means? A measurement is a fact about one configuration. Everything beyond that configuration is an argument you must still make, and label as one.
The path of this mechanism. Each stage can pass while the next fails, and a failure at any stage makes everything after it decorative.
| Stage | Part | What it secures | Typical failure |
|---|---|---|---|
| Legitimate observation | 1, 2 | The experiment is designed and the load is real | Laptop result, closed-loop load, coordinated omission |
| Informative observation | 3 | You recorded the right quantity | Means, averaged percentiles, peak throughput without an SLO |
| Real difference | 4 | The gap is more than noise and your own choices | One run, dropped outliers, sweeping 50 configurations |
| Explicable difference | 5 | You know where the time goes | Flat on-CPU profile of a waiting system |
| Transferable (or honestly bounded) | 6, 7, 8 | You know what the result licenses | Vendor slides, unpinned runs, extrapolation across a boundary |
What this mechanism forbids, in one line each. A number without its configuration. A comparison without a noise floor. A percentile without a composition rule. A capacity claim without a latency constraint. A conclusion without a control. An extrapolation without a named boundary.
The principles, in the order this mechanism met them
- The instrument is a system: the harness and collector obey the same laws as the system under test; when it shares fate with its subject it censors in the optimistic direction.
- Aggregation is lossy and irreversible: summarize after you know the question.
- Whoever chooses the conditions chooses the result: disclosure is the only audit surface.
- Systems are piecewise, so results are local.
- A control loop that includes the system under test measures the loop, not the system.
- The unit of measurement must match the unit of value.
- Selection is inference, and it consumes the evidence it selects on.
- Every instrument has a blind region, and the cost tends to hide in it: triangulate with instruments whose blind regions differ.
- A shared metric becomes a target and decays into a proxy for itself.
- Internal and external validity trade against each other; the exchange rate is what you pinned.
- An untested path defaults to broken: make rare paths not rare, on purpose.
- An explanation earns belief by predicting a second, different observation.
Older ideas that returned. Granularity mismatch (sampling rate versus event duration). Fixed cost per operation sets a natural batch size (here the operation is an experiment, and batch size is statistical power). Little's Law (inferring queue depth). Costs are conserved: every escape here moves a cost. Variance costs more than volume. Independence is assumed and almost never true (TPC-H, synthetic data).
Mechanisms to remember
- Predict before you measure; register the decision rule; stopping rules are part of the design.
- Open-loop load with intended-send-time latency; sweep up and down to find the knee.
- Distributions, not means; percentiles do not compose; report goodput and throughput at a latency SLO.
- Report the A/A noise floor, confidence interval and practical significance; confirm winners on held-out runs.
- Profile with both clocks (on-CPU and off-CPU), check IPC, apply USE to every resource, use differential profiles with a matched control.
- Read the schema and generator of a standard benchmark first; a result without a config dump is not a claim.
- Pin versions and seeds, publish raw results, automate reruns, state scale range and what the experiment cannot answer.
- In production: shadow traffic, canary, interleaved A/B (watch interference); fault injection with hypothesis, abort condition and bounded blast radius; report cost as a frontier.
Models and memory. Queueing models, the roofline and the USL are cheap experiments that explain; use a model to predict, measure two or three points to validate it, and interpolate. Record negative results: an internal corpus is survivorship-filtered just like the public one.
The place in the whole map. Every design claim about storage and data systems is an empirical claim with a validity range, set by measurements on specific hardware, workloads and prices. Methods here are permanent; conclusions are contingent on cost ratios that move. The experiment assumes its conditions resemble yours, it cannot verify that, and you pay.
The story in one paragraph. Ingrid started with "Tessera is 40% faster." She ended with a one-page decision: with group commit, Tessera improves on-sale p99 by 11% at the same cost across 1 to 30 TB, below the 15% rule written in advance, so do not ship for this on-sale. It states the mechanism (the flush path), the confirming prediction, the limits and the next experiment. Mateus could defend it because every sentence was a measurement with its configuration or a labelled argument.