Predictions, hypotheses and variables

Why predict and register before you run, and why does changing one variable at a time mislead?

Predict before you measure

This is the missing first step. An experiment's value is the uncertainty it removes. If you have no prior expectation, no outcome can surprise you, and nothing is learned. Prediction is also the cheapest error detector: a back-of-envelope calculation takes minutes, and the run takes hours.

The size of the discrepancy between your prediction and the measurement carries the diagnosis:

---

Hypothesis registered before the run

Look at how many small choices you make in any analysis: which metric, which aggregation window, where to cut off warm-up, what counts as an outlier, which of several runs to keep. Suppose there are ten innocent yes/no choices. That gives 2¹⁰ = 1,024 possible analyses, which means about a thousand possible stories. At least one of them will fit any dataset.

If you make these choices after seeing the data, the data selected the story, and the story was never tested against the data. Nothing about any single choice is dishonest. The arithmetic is what catches you.

The fix is the falsifiability test, in concrete form: before running, write down the number at which you abandon the claim. For example: "If the p99 improvement is under 15%, I will conclude the change is not worth the complexity." If you cannot write such a number, you are not running an experiment.

---

Independent variable, controls, and what is held fixed

One configuration is a fact. Two configurations that differ in one thing are evidence. A causal claim needs that contrast.

The trap: "one variable" is usually defined at the knob (the setting you turned), not at the mechanism (the thing inside the system that actually changes). Example: you raise the target file size. That single dial changes the request count, the compression ratio, the task parallelism and the per-task memory at the same time. You turned one dial and moved four mechanisms, and your result blames the dial for what any of the four did.

So control needs a causal graph, not a checklist. You must know what the knob touches downstream before you can say the contrast is clean. Whatever you failed to pin down is your implicit list of confounders, and you never see that list. Pinning is therefore cheap insurance against total invalidation. This is the conserved-cost idea in its usual form: you pay for rigour up front, or you pay later as a thrown-away week.

---

One variable at a time vs. factorial designs

This is an economic and combinatorial question.

OVAT is valid only if the response is additive, meaning the effect of each factor does not depend on the others. In systems that is almost never true. Compression level interacts with core count and network bandwidth. Batch size interacts with the memory limit. Parallelism interacts with skew. This is the independence assumption failing again, here in the response surface itself.

The consequence is sharp and widely ignored. OVAT finds the best value of each knob given that the other knobs stay at their starting values. That is a point on the axes through your starting configuration. It is not an optimum. Published tuning guides are overwhelmingly OVAT results, and they are wrong away from the path on which they were measured. That is not dishonesty. It is a design choice nobody disclosed. (This is why Ingrid's "batch size 512" advice failed on her machine.)

The escape: fractional factorial screening. A 2^(7−3) design tests seven factors, each at two levels, in sixteen runs (2⁷ = 128 would be the full set). It tells you which factors have main effects at all. Then you run a full factorial over the two or three survivors. The price is aliasing: in a fractional design, a main effect is confounded with interactions, so a factor can look important only because something it pairs with is. You buy a lower run count with lower resolution.