Measurement overhead and wall versus CPU time

When does instrumentation distort what it measures, and what does wall minus CPU time tell you?

Measurement overhead and the observer effect

Measuring has a physical price, and the price is not only CPU time. Instrumentation adds a cost per event. A fixed cost per operation sets a natural granularity: if the thing you instrument is cheap compared with the instrumentation cost, you have changed what you measure instead of observing it. Tracing every function call in a loop whose body takes 20 ns is not measurement.

The subtler costs are not the cycles:

• Instrumentation evicts cache lines and displaces the program's working set.
• It inserts optimization barriers, which stop inlining and vectorization, so the compiled code is not the production code.
• A timestamp call can be a system call, or an instruction that makes the CPU pipeline wait, which disturbs the pipeline.
• When instrumentation writes to shared counters, it creates contention that grows with core count. You can build a scalability ceiling inside your own measurement. (This is the Universal Scalability Law, USL, which says that contention and coordination cost limit scaling. Here it appears inside the instrument.)

Sampling is the usual escape, and its price is resolution: you cannot see events rarer or shorter than your sampling interval. A 99 Hz profiler takes a sample every 10 ms (1/99 s ≈ 10.1 ms), so it cannot resolve a 2 ms stall. This is not a small error. It is structural blindness. It is the granularity-mismatch problem: the instrument's time scale must be smaller than the event's time scale.

The honest procedure: measure the overhead directly. Run the workload with instrumentation off and then on, and report the difference as part of the result. If that difference is comparable to the effect you are chasing, the measurement does not exist.

---

Wall-clock vs. CPU time vs. off-CPU time

There are three clocks that measure three different things. Choosing the wrong one sends the whole investigation in the wrong direction.

• Wall-clock time is what the user experiences. It is the only clock that cannot be wrong about nothing: if the user waited, wall-clock time shows it.
• CPU time is the time the thread spent executing instructions.
• Off-CPU time is the gap: wall-clock minus CPU time. It is the time the thread was not running, because it was blocked on I/O, a lock or a queue, or was descheduled (the operating system gave the CPU to something else), or was paused by garbage collection (GC).

Most latency problems in data systems are off-CPU, and most profiling tools are on-CPU by default. This mismatch produces a common wrong conclusion. A CPU profile of a stalled system shows the CPU work that did happen and gives the cost to it. The optimization that follows makes the fast part faster, and the stall remains. A flat CPU profile of a slow system is a finding: it means the problem is not CPU. Yet people often read it as "nothing to optimize".

A useful rule before profiling: compute wall − CPU first. If that gap dominates, an on-CPU profiler cannot find the answer. The gap also separates two cases with opposite remedies. Runnable but waiting (the thread is ready, but there is no free core: scheduler contention, a capacity problem). Blocked (the thread waits for a dependency: lock, disk, network, a latency problem).

---

What this article forbids

• Reporting a mean as the summary of a latency distribution.
• Averaging percentiles across shards or windows, under any justification.
• Claiming a percentile finer than your sample count supports.
• Peak throughput as a capacity claim.
• Server-side counts as goodput.
• Any instrumented result without a measured overhead difference.
• On-CPU profiling as the first step on a system whose wall time exceeds its CPU time.
• Above all: aggregating at collection time in a form that cannot be merged.

Escapes and their prices

One principle