Micro versus macro, and benchmarketing

When does each kind of benchmark mislead, and why is a result without its configuration not a result?

Micro versus macro benchmarks, and when each misleads

Micro-benchmarks isolate one mechanism (for example, one hash-table insert, or one compression routine). They are the only way to observe a mechanism cleanly, and they mislead in a typical direction: by isolating, they remove the resource contention, cache pressure and interference that dominate in a real system. A data structure that benchmarks beautifully alone may evict the working set of everything around it from the cache. Micro-benchmarks also invite the measurement to disappear: a compiler that removes the unused result is the classic case, and it produces an implausibly good number (the 9,000,000 holds per second of Experiment design is the same failure: the work never happened).

Macro-benchmarks run a whole workload. They mislead in the opposite direction: they measure a composite and cannot attribute. An end-to-end improvement is real, but its cause is unknown. So it does not generalize to anything except workloads shaped like the one measured.

The solution is not to choose one. It is to require agreement. A micro result predicts what a mechanism does. A macro result confirms that it shows up at the system level. When the two disagree, the disagreement is the finding. It means that the isolated mechanism is not on the critical path, or that it competes with something the micro-benchmark removed. This is the same triangulation logic as the "blind region" principle, at the scale of benchmarks: micro cannot see interaction, and macro cannot see attribution.

Benchmarketing patterns

Benchmarketing means using benchmarks as marketing. The patterns below are not lies. Each is a defensible choice that is made again and again in one direction. That makes each one hard to rebut alone and damaging in total.

Tuned versus untuned baselines. The vendor's system is configured by its own experts. The competitor runs on default settings. Defaults are often an order of magnitude away from the best setting for a specific workload, so this alone can decide a comparison. Ask: who configured each system? Did anyone who knows the competitor review its configuration?

Query subsetting. The report shows 15 of TPC-DS's 99 queries. Each omission has a reason: unsupported syntax, a timeout, a dialect difference. But the omissions are not random. The queries that were dropped are mostly the ones the system handles worst. This is the principle "selection is inference, and it consumes the evidence it selects on", now as a publication practice.

Undisclosed caching. This is the most consequential pattern and the easiest to do by accident. A result cache, a page cache and a metadata cache each give large speedups on repeated queries, and Experiment design showed that cold versus warm can differ by orders of magnitude. If the run order or repetition count differs between the two systems, you are comparing cache states.

Unequal hardware. Different instance types, different hardware generations, different storage classes, different networks. Sometimes this cannot be avoided, when the systems have different deployment models. Then the honest move is to compare at equal cost and not at equal hardware, which is exactly the comparison that is usually missing.

Ignoring cost. This is the most common omission and the largest. Performance without price is half a result, and a system that is 2× faster on 4× the hardware is worse. The frontier from The right quantity (the set of best trade-offs between latency, throughput and so on) has a cost dimension. Reporting only the performance axis chooses the flattering projection, which is "whoever chooses the conditions chooses the result."

Scale selection. The test runs at the scale factor where your architecture's cliff has not been reached yet. A result at 100 GB tells you nothing about 10 TB if the working set stops fitting in memory somewhere in between ("systems are piecewise, so results are local").

The economic reason underneath all six. Publishing a result costs engineering time, and people do it only when the result is favorable. So the published set is filtered by outcome before you ever see it. Every vendor's benchmark shows the vendor winning. This is not because each vendor is dishonest. It is because the losing runs were never published. Survivorship in the collection of results is structural, not individual misconduct.

Why a result without a configuration dump is not a result

This is a strong claim, and it follows directly from Experiment design.

A measurement is a fact about a configuration. Without the configuration, the sentence has no subject. It is not a weak result or an unverified one. It is not a proposition: nothing is asserted that could be true or false.

So the requirement is not a courtesy. It is part of the definition. A complete disclosure includes: engine and version, every non-default setting, hardware generation and instance type, storage class, data generator and scale factor, run count and run order, cold or warm condition, and cost. And it must be dumped by a program, not described by a person, because a description records the intent, and the difference between intent and reality is where the interesting problems live.

The practical asymmetry matters as much as the principle. Producing a configuration dump is almost free for the author. Reproducing the result without one is impossible for the reader. So its absence is itself information. It usually means one of two things: the authors did not track their configuration (in which case they cannot rerun their own result), or disclosure would weaken the claim. This is "whoever chooses the conditions chooses the result" in its sharpest form: disclosure is the only audit surface, and the decision to disclose is made by the party with the interest.

What this article forbids

Transferring a benchmark result to a workload that stresses different mechanisms. Reading ClickBench as evidence about joins, or TPC-H as evidence about cardinality estimation under correlation.
Accepting a micro-benchmark as a system claim, or a macro-benchmark as an attribution.
Comparing a tuned system with a system on defaults.
Comparing performance without cost.
Treating a published comparison as a sample when it is a survivor.
Citing any number whose configuration is unknown.

Escapes and their prices

Escape	What it buys	What it costs
Run the standard yourself on your own hardware	Control	Engineering time, and the standard's own workload bias, which you have not escaped
Build a custom benchmark from your real workload	Relevance	Incomparability: nobody else's number means anything against yours, and you lose the shared knowledge that standards were made for
Auditable full-disclosure reports	Trust	Effort, and publishing your own weaknesses, which is why few do it
Compare at equal cost, not equal hardware	Fairness across deployment models	The result depends on prices and so is perishable

One principle

Case, resolvedConstructed case · Ingrid at Bellwether Tickets

3× faster on a standard benchmark

A vendor slide claimed Tessera was 3× faster than the competition on a standard benchmark, and nothing on it was false. Where it started: Standard benchmarks →

Ingrid sends Chidi five questions, politely, before the next meeting.

Which benchmark, and what does it stress? It is a YCSB-style point-access workload: it stresses key-value reads and writes under skew and cannot stress joins or planning. Bellwether's hold service is point access, so this one is relevant. That is good for Chidi, and she says so.
Who tuned the competitor? The competitor ran on defaults. Tessera was tuned by Tessera engineers.
What was the cache state and run order? Tessera ran last, after the data was warm. The competitor ran first, cold.
What hardware and what price? Tessera used 3× the memory. At equal monthly cost the gap shrinks.
Where is the configuration dump? There is none, and nothing on the slide can be reproduced.

Chidi is honest and sends the full configuration within a day. Ingrid reruns the same workload with both systems tuned, in the same cache state, at equal cost. The "3×" falls to about 1.15×. This agrees with her own tests. The slide was not false. It was a survivor of many choices, all in one direction. She now uses the standard only as one input, and she builds a replay of Bellwether's own on-sale traffic as the main benchmark.