Catch a benchmark that did no work (30 minutes)

Measure your machineNeeds C compiler, Python

Problem

A benchmark measures whatever the machine actually did, which is not always what the code says. An optimizing compiler removes work whose result is never used, so a loop that "writes" a value a hundred million times can take no time at all. The defence is a prediction made before the run: if the measured rate is orders of magnitude beyond what the hardware could deliver for the claimed work, the work did not happen.

Write a loop that "writes" a value 100 million times and times it, compiled with optimizations (cc -O2). Then (1) do a back-of-envelope prediction: one fsync-backed write on your disk costs about 0.1 to 1 ms, so the limit is roughly 1,000 to 10,000 durable writes/s. (2) Compare the printed rate to your prediction. (3) Make the result observable (write to a volatile variable) and rerun. Record the prediction before you run.

The question. What rate does each loop report, and how does it compare with what a durable write could achieve?

Predict

The loop whose result is never used, compiled at -O2: what does it measure?

Billions of operations a secondAbout what a durable write allowsZero time: the loop is gone

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

catch-a-benchmark-that-did-no.py54 linesDownload

View the code

catch-a-benchmark-that-did-no.py

# /// script
# requires-python = ">=3.10"
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Catch a benchmark that did no work.

A C loop "writes" a value 100 million times and never uses it, compiled with
-O2; then the same loop with the result made observable (a volatile store).
Both rates are compared with the back-of-envelope ceiling for durable writes.
The C program is compiled and run by this script (needs cc).
"""
import argparse, json, os, subprocess, tempfile

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--iter", type=int, default=100_000_000, help="Loop iterations.")
p.add_argument("--opt", default="-O2", help="Compiler optimization flag.")
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()

C = r"""
#include <stdio.h>
#include <time.h>
int main(void){
  struct timespec a,b; clock_gettime(CLOCK_MONOTONIC,&a);
#ifdef OBSERVE
  volatile int x = 0;                         /* every store must happen */
#else
  int x = 0;                                  /* result never used */
#endif
  for (long i = 0; i < ITER; i++) x = (int)i;
  clock_gettime(CLOCK_MONOTONIC,&b);
  double s = (b.tv_sec-a.tv_sec)+(b.tv_nsec-a.tv_nsec)/1e9;
  printf("%.6e %.9f\n", ITER/s, s);
}
"""
d = tempfile.mkdtemp(); src = os.path.join(d, "hold_bench.c"); open(src, "w").write(C)
rows = []
for name, flags in (("result never used", []), ("result observable (volatile)", ["-DOBSERVE"])):
    exe = os.path.join(d, name.split()[1])
    subprocess.run(["cc", args.opt, f"-DITER={args.iter}L", *flags, src, "-o", exe], check=True)
    rate, sec = map(float, subprocess.run([exe], capture_output=True, text=True, check=True).stdout.split())
    rate = None if sec == 0 else rate                                      # zero elapsed time: the loop was deleted
    rows.append({"variant": name, "ops_per_s": rate, "seconds": sec, "ns_per_op": sec / args.iter * 1e9,
                 "times_disk_ceiling": None if rate is None else rate / 10_000})
print("prediction: one fsync-backed write costs 0.1-1 ms, so 1,000-10,000 durable writes/s")
for r in rows:
    if r["ops_per_s"] is None: print(f"{r['variant']:<30} measured {r['seconds']} s for {args.iter:,} iterations: the compiler removed the loop")
    else: print(f"{r['variant']:<30} {r['ops_per_s']:.3e} ops/s  ({r['ns_per_op']:.3f} ns/op, {r['times_disk_ceiling']:,.0f}x the 10,000/s ceiling)")

if args.json:
    with open(args.json, "w") as f:
        json.dump({"tables": {"variants": rows}}, f)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run catch-a-benchmark-that-did-no.py
uv run catch-a-benchmark-that-did-no.py --help     # every knob
```

Knobs
Knob · default	What it changes
--iter100,000,000	Loop iterations.
--opt-O2	Compiler optimization flag.

Our run

One loop, two builds

One loop, two builds
Variant	Measured time (s)	ns per op	ops/s
result never used	0	0	—
result observable (volatile)	0.054	0.543	1.84B

Reference runApple M2 · 8 cores · 8 GB

Output · uv run catch-a-benchmark-that-did-no.py

prediction: one fsync-backed write costs 0.1-1 ms, so 1,000-10,000 durable writes/s
result never used              measured 0.0 s for 100,000,000 iterations: the compiler removed the loop
result observable (volatile)   1.843e+09 ops/s  (0.543 ns/op, 184,325x the 10,000/s ceiling)

macOS 26.6.2 · Python 3.13.3 · run 2026-10-20

What we found

The loop whose result was never used measured 0 seconds: the compiler deleted it, so the rate is not billions but undefined. With the result stored to a volatile variable it ran at 1.84B operations per second (0.543 ns each), still 184K times the 10,000-per-second ceiling. That run did real work, but the wrong work: a store to memory, not a durable write.

Read it

The prediction is what catches both failures. A zero time says the benchmark measured nothing; a rate far above the physical ceiling says it measured something other than the claim. Writing the expected order of magnitude down before running is the cheapest benchmark validation there is.