Threats to validity and scale honesty

Which confounders remain, and what may a 1 TB result say about 20 TB?

Threats to validity

Reproducibility is now settled. The next question is extrapolation, and the threats to validity are the specific ways an extrapolation fails.

Confounding. Something varied together with the treatment. Experiment design addressed it by design, but some always remains. The honest move is to name the confounders you could not remove, and not to claim there were none.

Hardware-generation dependence. The relative cost of CPU, memory bandwidth, storage and network changes with each hardware generation, and many design choices are bets on one particular ratio. If a result says a layout wins, it was true against a specific ratio of memory latency to storage latency. When fast NVMe SSDs narrowed the gap to memory, they invalidated a decade of designs that assumed disk was 100,000× slower. This is the most under-appreciated threat, because it makes old results wrong and not merely old, and the wrongness is invisible in the result itself.

Survivorship in the workload. The queries you measure are the queries people run, and people only run queries that finish. Queries that would have been slow were rewritten, abandoned or never written. So a trace of a production workload systematically understates the weaknesses of the system. It is a record of what this system is good at, shaped by users who adapted around it. This is why migrating to a faster engine often reveals workloads that nobody knew existed: the constraint that had suppressed them is gone. It is "selection consumes evidence" again, here done by users and not by the experimenter.

Construct validity. This is the deepest threat. Did you measure the thing you really care about? Query latency is a proxy for analyst productivity. p99 is a proxy for user experience. TPC-DS is a proxy for analytic workloads. Every proxy is valid within a range and drifts outside it. This is "the unit of measurement must match the unit of value": the gap between the unit measured and the unit valued is where confident conclusions go wrong.

---

Scale honesty

This is the extrapolation that fails most often, and it has a mechanism, not only a warning.

Systems are piecewise, and the pieces are separated by discontinuities:

• the working set grows beyond cache, then beyond memory, then beyond a single node;
• the partition count crosses a threshold in a scheduler or a metadata service;
• the request rate crosses a per-prefix limit in a cloud service;
• the data volume crosses the point where a broadcast join (copy the small table to every node) becomes a shuffle join (redistribute both tables by key).

Each boundary is a change of algorithm, not a change of slope. A curve measured entirely inside one piece contains no information about the next piece. Fitting a straight line through it and extending the line is not cautious. It is a claim that no boundary exists, which is the strongest possible claim, and it always goes unstated.

Both directions are common:

• Upward: "Linear up to 1 TB, so fine at 100 TB." This ignores every boundary above 1 TB.
• Downward: a result at 100 nodes says little about 3 nodes, where the coordination overhead matters in a different way and constants that were negligible are not.

The discipline is to state the tested range and to argue explicitly about generalization by naming the mechanisms that would change: which resource saturates next, which algorithm switches, which limit is near. That argument can be tested and is useful. A blanket "results may not generalize" is neither.

---

Stating what the experiment cannot answer

This is the closing discipline, and it is the cheapest valuable one here.

Every experiment has a scope, and the scope can be known in advance: it is the complement of the configuration you pinned. The pinning that bought internal validity is exactly what gave away external validity. This is the trade that this article describes. Experiment design held things fixed to make the contrast clean, and each fixed thing is now a dimension that the result says nothing about.

Why write the limits down and not leave them implied? Because of an asymmetry in cost. The author knows the limits at almost no cost. The reader cannot recover them at any cost, and when they are missing the reader will assume the widest scope, because that is what an unqualified claim means. This is "disclosure is the only audit surface" once more, now for scope and not for configuration.

A practical version does more than a limitations paragraph: state the next experiment. For example: "This measures cold start on one instance type. The open questions are warm behaviour and behaviour above 10 TB." That turns a limitation into a research direction, and it tells the reader clearly when they are outside the evidence.

---

What this article forbids

• Unpinned versions, unfixed seeds, or trusting a container to pin the machine.
• A chart as the only artifact.
• Manual reruns in any study that needs more than one comparison.
• Extrapolation across a boundary you did not measure, in either direction.
• Treating a production trace as an unbiased sample of demand.
• Any result presented without a stated scope.
• The subtle one: presenting an argument about generalization in the same voice as the measurement, so the reader cannot tell which is which.

Escapes and their prices

Nothing in this article removes a cost. Each escape moves it. "Costs are conserved, never removed" holds as firmly at the top of the stack as at the bottom.

One principle