Why standard benchmarks exist, and what each stresses
What do standard benchmarks buy, and which parts of a system does each put on trial?
A different kind of problem
The first five articles were all about making your own measurement trustworthy. This article is about measurements produced by someone else. The dominant constraint here is no longer physical or statistical. It is economic. A standard benchmark is a shared artifact with money attached to its results. Once a metric is a target, the optimization pressure lands on the metric, not on what the metric was meant to represent.
One constraint disappears: you no longer control the conditions. One constraint appears: you can audit only what was disclosed, and the person who discloses chose what to disclose.
A benchmark is a workload someone else chose, and choosing the workload is most of choosing the winner.
Why standard benchmarks exist at all
It is worth stating the purpose before the criticism, because the criticism is only useful against a clear purpose.
Custom benchmarks cannot be compared across organizations. Everyone measures a different thing, and no shared knowledge builds up. A standard benchmark fixes the workload, so results can be compared across vendors and across years. That is a real and large benefit. The cost is that a fixed workload is a fixed opinion about what matters, frozen at the date it was written. Systems then evolve toward it.
This is the basic tension: a benchmark is useful because it is stable, and it decays because it is stable. Its representativeness is highest on the day it is published and then falls, while its authority does the opposite and grows as more papers cite it. The gap between a benchmark's credibility and its relevance widens with age.
What each standard actually stresses
The useful question is not "which benchmark is best?" It is "which mechanism does this one exercise?" A result transfers only to workloads that stress the same mechanisms.
| Benchmark | Shape | Mainly stresses | Structurally cannot stress |
|---|---|---|---|
| TPC-C | OLTP (many small transactions, like orders) | Concurrency control, lock and latch contention (a latch is a short-term lock on an internal structure), the write-ahead-log and commit path, row-level access | Scan performance, plan quality, columnar layout |
| TPC-H | Analytic, 22 queries, simple schema | Join algorithms, scan throughput, basic optimizer choices | Cardinality estimation under correlation, concurrency, semi-structured data |
| TPC-DS | Analytic, about 100 queries, snowflake schema (a central fact table linked to branching dimension tables), skewed data | Optimizer quality, join-order search, complex predicates, window functions | Ingestion, streaming, mutability |
| YCSB | Key-value, point access | Throughput and latency under a read-write mix and skew, replication behaviour | Anything that needs joins, transactions or query planning |
| ClickBench | Single-table analytic, flat schema | Scan speed, filtering, aggregation, compression | Joins entirely, because the schema has none |
Two things follow immediately.
First, ClickBench has one flat table. A result on it says nothing about join-heavy work, which is most real analytics. It measures the scan path (storage format, compression, scan speed) and nothing about the planner.
Second, TPC-H has independent, uniformly distributed columns. This systematically flatters cardinality estimation, because the optimizer's assumption that columns are independent is true in that data. (Real data has correlated columns, for example city and postal code. When the optimizer assumes independence it multiplies selectivities and under-estimates row counts.) TPC-DS was written partly to fix this, which is why it separates engines that TPC-H ranks equally. This is the "independence is assumed everywhere and true almost nowhere" rule once more: the benchmark's synthetic independence hides the exact weakness that matters in production.
The general rule: read the schema and the data generator before you read the results. They decide which mechanisms can possibly be exercised. No amount of method can fix a workload that structurally leaves out your bottleneck.