Sampling, instrumentation and hardware counters
When should you sample and when instrument, and what do hardware counters say the CPU waited for?
Sampling versus instrumentation
This is the same trade-off that appears with summaries (sketches and histograms: compact structures that approximate a stream of values), here in a third form.
Sampling has a cost proportional to the sampling rate, and it does not depend on the event rate. A hot loop that runs a billion times costs the same to profile as an idle thread. So sampling disturbs the program evenly and degrades gracefully. Its error is statistical: rare events may be missed entirely, and your confidence in a frame's width follows from its sample count (see the worked example on the previous page).
Instrumentation means adding code that records every event, for example a timer at the start and end of a function. Its cost is proportional to the event rate. It is exact for the events it covers and blind to everything else. It disturbs the program unevenly: the hottest paths pay the most, so the profile it produces is distorted in a particular direction, and hot paths look hotter than they are. Also, instrumentation inserts barriers that stop compiler optimizations, so the instrumented binary is not the binary you deploy.
The rule for choosing follows from which failure you can accept:
- Sampling answers "where does the bulk of the time go?" It cannot answer "why was this one request slow?"
- Instrumentation, and especially tracing (recording the path of one request through the system), answers the second question. It cannot afford to answer the first at full coverage.
That is why production systems sample traces at a low rate. And that low rate makes the traces useless for diagnosing rare events, because the events you most want are exactly the ones that sampling throws away.
Adaptive schemes are the partial escape: sample more when requests are slow, or start a trace when a tail event is detected. The price is that the sampling probability now depends on the outcome, which biases every aggregate computed from the samples. You bought rare-event visibility with a dataset that you can no longer average.
Hardware counters
Hardware counters exist because the layers above cannot see why a CPU cycle was spent. An instruction counts as finished (the CPU says it "retired" the instruction) whether it did useful work or waited for memory. Only special counters in the CPU, called performance monitoring counters, can tell the difference. On Linux the tool perf stat reads them.
IPC (instructions per cycle) is the headline number. A modern core can finish several instructions in one clock cycle. An observed IPC well below 1 means the core spent most of its cycles stalled (waiting), and the counters can name what it waited for. This is the hardware-level version of the wall-minus-CPU check. CPU time itself divides into working and stalling, and "100% CPU" says nothing about which one it was.
The causes then split into separate counters:
| Counter | What it means | Typical remedy |
|---|---|---|
| Cache misses at each level (L1, L2, L3) | Data was not in the fast memory near the core. A miss that goes all the way to DRAM (main memory) costs about 100× an L1 hit. | Better layout, smaller working set. A scan-heavy workload is usually bound by memory bandwidth, not by computation (the "roofline" distinction: either memory or arithmetic is the limit). |
| Branch mispredictions | The CPU guessed wrongly which way an if goes and flushed its pipeline. | Branchless and vectorized code. |
| TLB misses | The working set is scattered over too many memory pages, so address translation is slow. | Huge pages. |
| Memory stall cycles | The direct fraction of cycles spent waiting for memory. | Move fewer bytes. |
Why this matters in data systems specifically: many well-known optimizations (columnar layout, dictionary encoding, vectorized execution, cache-conscious hash tables) are justified by these counters and by nothing else. At the level of the query plan they are invisible: the plan looks identical with or without them. The counters are the only instrument that shows the mechanism working.
The limits are real:
- Counters are per core. To attribute them to a logical operator you must build the correlation yourself.
- Multiplexing: a CPU has only a few counter registers, so you cannot measure many events at the same time. The profiler switches between events in time slices, and each reported count is an extrapolation.
- On shared or virtualized hardware, counters are often unavailable, or they measure your neighbours as well as you.