Interview questions
Designing the experiment
1Why is a benchmark result a belief, not a fact?
It is a fact only about one configuration: the hardware, version, settings, data and load it ran with. Any claim beyond that needs stated assumptions.
Read moreWhat kind of thing is a measurement?
2What should you do before running a benchmark?
Write down a prediction from a back-of-envelope calculation, so that an impossible result is caught as a bug. For example, 9 million durable writes per second on a disk limited to about 10,000 per second.
Read morePredict before you measure
3What does it mean to pre-register a decision rule?
Writing down, before the run, which result will make you ship or not, for example "p99 improves by at least 15% at equal cost". Then you cannot choose the criterion after seeing the data.
Read moreHypothesis registered before the run
4Why is testing one factor at a time risky?
It misses interactions between parameters, such as batch size and thread count. Factorial designs test the combinations.
5Why must you record versions and configuration with every result?
Whoever chooses the conditions chooses the result. Without a dump of versions, settings and hardware, nobody can audit or reproduce it.
Read moreVersion and config pinning
Generating load
6What is a closed-loop load generator, and why is it dangerous?
It has a fixed number of users, each waiting for a response before sending the next request. By Little's Law, the request rate is users ÷ response time, so the rate falls as the system slows, and the test can never reach overload.
Read moreOpen-loop vs. closed-loop
7What is coordinated omission?
A load generator that waits for each response leaves out the requests it would have sent during a stall, so the worst samples are never recorded. Fix it by measuring latency from the intended send time.
Read moreCoordinated omission
8Why use exponentially distributed think time between requests?
A constant think time makes clients synchronize into artificial bursts. Random, exponentially distributed think time models independent arrivals.
Read moreThink time, ramp-up and steady state
9How do you find a system's saturation point?
Sweep the arrival rate up in fine steps and back down again. Look for the knee, and for hysteresis on the way down, where the system recovers more slowly than it degraded.
Read moreSaturation discovery
Measuring the right quantity
10Why can you not average p99 latency across shards?
Percentiles are not linear, so their average has no meaning. Merge the histograms and compute the percentile last.
Read morePercentiles do not compose
11How does fan-out amplify tail latency?
If a request waits for N independent backends, each fast with probability p, it is fast only with probability p^N. With 100 backends and p = 0.99, about 63% of requests hit at least one slow node.
Read morePercentiles do not compose
12Why is 'peak throughput' a poor capacity claim?
It is measured past the knee, where latency has already exploded. Report throughput at a fixed latency target instead.
Read moreThroughput at a fixed latency SLO
13What is goodput?
Work that was useful to a client, counted at the client against the original request. During a retry storm, throughput can be at its maximum while goodput is near zero.
Read moreGoodput vs. throughput
14What does a flat CPU profile of a slow system tell you?
The time is spent off the CPU: waiting on locks, I/O or queues. Compute wall-clock time minus CPU time before choosing a profiler.
Read moreWall-clock vs. CPU vs. off-CPU time
Statistics for benchmarks
15What is an A/A test for?
Running the same configuration against itself measures the noise floor: how much the result varies when nothing changed. A difference smaller than that is not a result.
Read moreThe noise floor
16Why is a whole run, not a single operation, the unit of sampling?
Operations inside one run share the same environment, so they are not independent. Millions of them give precision about that one execution only; the number of runs sets what you can conclude.
Read moreRun counts and the two variances
17Why does the winner of a tuning sweep disappoint in production?
Regression to the mean: the winner was selected for being extreme, and part of its score was luck. Confirm it with a fresh, held-out run.
Read moreRegression to the mean
18What is optional stopping, and why is it a problem?
Running until the result looks significant. It inflates false positives, so fix the number of runs in advance.
Read moreThe stopping rule
Profiling and attribution
19Why can an on-CPU flamegraph mislead?
It samples only threads that are running, so time spent blocked is invisible. Check wall-clock minus CPU time first; if the gap dominates, use an off-CPU profile.
Read moreFlamegraphs: on-CPU and off-CPU
20What does the width of a flamegraph frame mean?
Inclusive cost: the function plus everything it called. The function's own time is the part not covered by frames above it.
Read moreFlamegraphs: on-CPU and off-CPU
21Sampling or instrumentation: when do you use each?
Sampling finds where the bulk of time goes, with uniform overhead. Instrumentation or tracing explains one slow request, but costs something per event.
Read moreSampling versus instrumentation
22Why is low-rate trace sampling bad for rare events?
The rare slow requests are exactly what a low sampling rate discards. Adaptive sampling helps, but biases averages.
Read moreSampling versus instrumentation
23What does an IPC (instructions per cycle) well below 1 tell you?
The core is stalled most of the time, usually waiting for memory. "100% CPU" does not say whether the core was working or waiting.
Read moreHardware counters
24What are the three USE measurements, and why do you need all three?
Utilization, saturation and errors, for each resource. They do not move together: moderate utilization with saturation means bursty or serialized work.
Read moreThe USE method
25Why is summing the time of each query operator wrong?
Pipelined operators run overlapping in time, so the sum exceeds wall-clock time. The critical path decides the duration.
Read morePer-operator profiles
26When is comparing two flamegraphs (a differential flamegraph) valid?
When both were taken under the same load, with enough samples per frame, and with comparable stack shapes (inlining can change them).
Read moreDifferential profiling
Standard benchmarks
27What does TPC-H flatter, and why?
Cardinality estimation. Its columns are independent and uniformly distributed, so the optimizer's assumption that columns are independent happens to be true.
Read moreWhat each standard stresses
28Why can a ClickBench result not predict performance on join-heavy work?
Its schema is a single flat table, with no joins.
Read moreWhat each standard stresses
29Name common benchmarketing patterns.
A tuned system against a default baseline; reporting a subset of queries; undisclosed caching; unequal hardware; ignoring cost; and choosing a flattering scale.
Read moreBenchmarketing patterns
30Why is a result without a configuration dump not a result?
A measurement is a fact about a configuration. Without the configuration, nothing testable is claimed.
Reproducibility and validity
31Does running in a container make a benchmark reproducible?
It pins only user space. The kernel, CPU generation, microcode and hardware still vary.
32Why publish raw results, not just charts?
A chart fixes every choice of aggregation. Raw data lets readers recompute percentiles and spread.
33Why does a one-command rerun matter for what you can know?
The cost of a rerun sets how many runs you can afford, and so your statistical power and your ability to run A/A tests and held-out confirmations.
Read moreOne-command reruns
34What is the risk of extrapolating a curve measured at small scale?
Systems behave piecewise. A curve inside one regime says nothing about where the next boundary is, such as data no longer fitting in memory.
Read moreScale honesty
35Why is a random train/test split wrong for temporal or grouped data?
It assumes rows are interchangeable. With temporal data, the future leaks into training; with grouped data, one customer ends up on both sides.
Read moreSplits
36Why derive the train/test split from a hash?
A hash of a stable key gives the same assignment on every rerun, so adding rows does not reshuffle old test rows.
Read moreStable splits from a hash
Measuring in production
37What is interference between the arms of a live A/B test?
The two groups share caches, databases and connection pools, so a heavy treatment slows the control too, and biases the comparison.
Read moreExperimenting in production
38What makes a fault-injection run an experiment rather than a stunt?
A hypothesis about steady state that could be proven wrong, one injected fault, a declared condition for aborting, and a limited blast radius.
Read moreChaos testing: the method