Plan a benchmark
You are about to benchmark something that will decide a purchase, a migration or a tuning change: a new engine, a storage format, a configuration. A benchmark result is a fact about one configuration only, and it is easy to measure the wrong thing with great precision. How do you plan a benchmark whose result you can trust and act on?
You will end with a benchmark whose result can decide something: a prediction, a registered decision rule, a recorded configuration, a measured noise floor and a one-command rerun.
Work through these before the first run.
Work through it
Predict the result first
Write down the expected range from a back-of-envelope calculation: device limits, bandwidth, the arithmetic of the workload. An impossible result, such as 9 million durable writes a second on a disk limited to about 10,000, is then caught as a bug instead of celebrated.
Register the decision rule
Before running, write down which result means ship and which means do not: for example, "adopt only if p99 during peak improves by at least 15% at the same monthly cost, at 1×, 3×, 10× and 30× load". Then the criterion cannot drift after you see the data.
Pin and record the environment
Record versions, settings, kernel, instance type and hardware with every run. Watch for burstable instances that run on CPU credits, and state whether each run starts cold or warm. Whoever chooses the conditions chooses the result; without the record, nobody can check it.
Use the real workload, at several scales
Use your own key distribution, data and queries, not a standard benchmark whose data is uniform where yours is skewed. Measure at several scales, because systems change behaviour at boundaries, such as data no longer fitting in memory.
Generate load that cannot hide overload
A load generator with a fixed number of users, each waiting for a reply, slows down as the system slows, so it can never reach overload (closed loop). Send requests at a set rate regardless of replies (open loop), measure latency from when each request should have been sent, and sweep the rate up and back down to find the knee.
Measure the noise floor
Run the same configuration against itself (an A/A test). A difference smaller than that spread is not a result.
Fix the run count and the run order in advance
Decide how many runs before starting, so you cannot stop when the answer looks good. Interleave A and B runs in random order, so drift in the environment hits both. Treat each run, not each operation, as one sample. Declare any rule for dropping outliers before the runs.
Test interactions, not one knob at a time
Changing one setting at a time misses interactions, such as batch size with thread count. Screen many factors in a few runs, then test the important ones in combination.
Confirm the winner with a fresh run
The best of many configurations was partly lucky. Re-run it, held out from the search, and use that number.
Make it rerunnable, and publish the raw results
One command should rerun everything, so more runs are affordable. Keep raw results, not only charts, so others can recompute percentiles and spread.