Publishing configs and one-command reruns

What must be published for a reader to check a result, and why does rerun cost set your rigor?

Publishing configs, generator and raw results

The artifacts you publish have very different value, so the order matters.

A chart is the least valuable output, and it is the one most often published. It contains every aggregation choice from The right quantity and cannot be undone ("aggregation is lossy and irreversible"). A reader cannot compute a different percentile, apply a different outlier rule, or check the spread between runs. Raw results (per-operation observations, or at least per-run observations) let a reader ask questions the author did not.

The configuration makes the result a proposition at all (Standard benchmarks). The workload generator makes the result reproducible in kind when the original hardware is not available, which is the common case. Together they form a hierarchy:

The economic reality is worth stating without moralizing. Full publication is rarely returned in kind, costs real effort, and exposes weaknesses. So it is rare, and the rarity is a structural result of the incentives and not a failure of individual character. The practical consequence: the presence of full artifacts is strong evidence of good faith, because the cost of producing them is paid precisely by people who expect scrutiny to help them.

---

One-command reruns

This looks like tooling hygiene. In fact it is an epistemic mechanism (a mechanism that decides what you can know).

The cost of a rerun decides how often you rerun. How often you rerun decides how many of the disciplines in Statistical hygiene you can afford. A/A noise floors, interleaved ordering, held-out confirmation runs and seed sweeps each multiply the number of runs. If one rerun costs a day of manual setup, none of them will happen, and the method quietly degrades to a single run reported as fact.

So automation is not a convenience. It is the thing that makes rigor cheap enough to practice. This is the rule "a fixed cost per operation sets the natural batch size" in a new area: the operation is an experiment, and the batch size is your statistical power. When you lower the fixed cost, the number of runs you can afford goes up. It is the same argument as group commit (many commits share one disk flush) and the same as batching on the network.

There is a second effect. A scripted experiment is a complete specification of itself. It cannot leave out a configuration step the way prose can, because leaving out a step breaks the script. The automation is a configuration dump that is checked by execution.