Measurement & Experimental Method
What would have to be true for this number to mean what I think it means?
Who this mechanism is for: a developer who has run a benchmark, read a vendor comparison, or looked at a latency dashboard, and who wants to know when such a number can be trusted. You do not need any statistics or database background. Every technical term is explained the first time it appears, and again in the glossary at the end. This mechanism is about method, not about one product: it applies to databases, web services, queues, compilers and cloud bills alike.
The story we will follow
Ingrid is a platform engineer at Bellwether Tickets, a company that sells concert and sports tickets. Bellwether's hardest moment is the Saturday 10:00 on-sale: a famous band releases 80,000 seats, and for a few minutes the site receives more traffic than in the rest of the week. The key component is the seat-hold service. When a fan clicks a seat, the service stores a "hold" (a reservation that lasts five minutes) in a storage engine, and later confirms it or releases it.
In eight weeks there is another big on-sale. Mateus, the engineering manager, must decide whether to replace the current storage engine ("Engine A", tuned in-house) with a vendor product called Tessera DB. He wants one answer: "Is it faster, yes or no?" Yuki owns the analytics lake and the latency dashboards. Chidi, the Tessera sales engineer, will appear later with slides. Everyone brings numbers. Ingrid's job is to find out which numbers can be believed.
Ingrid will meet a problem at the start of each article. Each one looks like a bug: a result that changes from day to day, a test that says "stable" while production fails, a dashboard that looks healthy while customers complain. Each one is really a rule of method. At the end of each article we go back to Ingrid and solve her problem.
This is the main idea of this mechanism:
How this mechanism is organized
- 1
Experiment design
How do I design an experiment so that one explanation survives and the others die, including tests of failure and of consistency?
Case40% faster on Monday, slower on Thursday
- 2
Load generation
How do I apply load without letting the system under test vote on how much it receives?
Casep99 of 40 ms in the test, collapse at the on-sale
- 3
The right quantity
Which quantity do I record, and why can some summaries never be undone?
CaseMean latency 12 ms, and fans timing out
- 4
Statistical hygiene
Is the difference real, or is it noise plus my own choices?
Case+14% in the sweep, +4% in production
- 5
Attribution and profiling
Where does the time actually go?
CaseSlower at p99, and a flat profile
- 6
Standard benchmarks
What do other people's published numbers mean?
Case3× faster on a standard benchmark
- 7
Reproducibility and validity
What does a result allow me to believe about situations it never touched?
CaseAnother region, another answer
- 8
Beyond staging
What can only an experiment on the real system show, what does it cost, and when is a measured difference a reusable result?
CaseStaging is the optimistic case
After that: a summary, hands-on labs, interview questions and a glossary.
One request: do not skip the small calculations. This mechanism uses many short worked examples (for example, "a 100-way fan-out is slow 63% of the time"). The arithmetic is the real content.
What do I do, and what does each option cost?
Can I see it myself?
Run your own config dump, seed sweep and scale ladder
One command: dump the configuration, generate data from a seed, sweep 1x to 30x, write raw results; random lookups into 8 to 240 MB, five seeds.
MeasurePython with NumPyClosed-loop vs. open-loop, and coordinated omission (45 minutes)
A server slows from 40 ms to 400 ms under a closed-loop and an open-loop client; then a one-second freeze measured with and without intended-time correction.
MeasurePythonKill a node and measure the recovery
Three instances behind a health-checked proxy; kill the primary under steady load, three times, against the hypothesis "below 1% errors within 10 s".
Break itPythonMean, percentile averaging and fan-out in 20 lines (30 minutes)
One million latencies, 1% of them slow: the mean, the percentiles, an average of shard p99s, and a fan-out of 100 calls.
SimulatePython with NumPyWall time minus CPU time
10,000 small writes with an fsync after each: wall time against CPU time, then one flush per 100 writes; with fsync and with macOS F_FULLFSYNC.
MeasurePythonMeasure your noise floor and your winner's curse (45 minutes)
An A/A spread over 10 identical runs, a "tuning sweep" of 30 identical configurations, and a blocked against interleaved A/B under background load.
MeasurePython with NumPyA micro-benchmark that disappears, and a sampling blind spot
A billion x*x computed and discarded, then summed; and a 2 ms sleep once a second profiled at 99 Hz and 999 Hz.
MeasureC compiler, PythonDifferential profile of two builds
Profile a JSON-parse-and-insert loop before and after adding a lock, diff the profiles frame by frame, then repeat with one side under extra load.
MeasurePythonCatch a benchmark that did no work (30 minutes)
A C loop that "writes" a value 100 million times, with the result unused and then observable, against the ceiling for durable writes.
MeasureC compiler, PythonRandom split versus group split versus hash split
50,000 rows for 500 couriers with personal biases: a random split, GroupKFold by courier, and a hash split; then 10% more rows and how many test rows move.
MeasurePython with scikit-learn
Can I reason about it under questioning?
Interview questions
Short questions on measuring data systems, such as experiment design, load generation, percentiles and goodput, statistics for benchmarks, profiling, standard benchmarks and reproducibility, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on measuring data systems, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Generality vs. specialization
For example: general engines vs. purpose-built structures.
Latency vs. throughput
For example: batching, group commit, micro-batch.
Exactness vs. approximation
For example: sketches, sampling, ANN.
Isolation vs. utilization
For example: multi-tenancy, resource pools, spot capacity.
Early binding vs. late binding
When structure is committed; determines where the failure surfaces, at write time or read time.
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Guarantees have scope
Atomicity is bounded by the shared commit substrate.
Absence is not evidence
Absence of an event is not absence of a failure.
Independence is assumed
Independence is assumed everywhere and is true almost nowhere.
Fixed cost per operation
Fixed cost per operation implies a natural batch size.
Sound approximations
When exactness is too expensive, use a sound approximation.
Show all 11 principlesShow fewer
Queues explode near full load
Response time explodes as utilization nears 100%.
Variance costs more than volume
Variance costs more than volume.
Which way the error points
Hidden state breaks replay
A replayable pipeline is a pure function of its retained inputs.
Local optimizations compose badly
Identical mechanisms at adjacent layers do not add up. They interfere.
Undeclared dependencies
An optimization that succeeds becomes an undeclared dependency, with leverage 1/(1 − h).