From a number to a map: flamegraphs

What do on-CPU and off-CPU flamegraphs show, and what can a handful of samples not show?

CaseConstructed case · Ingrid at Bellwether Tickets

Slower at p99, and a flat profile

Ingrid has fixed her method. The load is open-loop, the quantity is p99 at a fixed latency target, and the comparison has a noise floor. The corrected benchmark still shows a clear result: on the hold-write path (the code that stores a seat hold), Tessera DB is slower than Engine A at p99, and the difference is far larger than the noise.

Mateus asks the natural question: "Why?" Ingrid takes a CPU profile of Tessera under load. It is almost flat. No function stands out. The CPU is not even busy: the cores are at 35%. She has a real difference and no idea where it lives.

This is not a failed profile. The profile is accurate, but it looks at the wrong part of the time. To see which part, we need to understand how profilers and counters work, and what each one cannot see.

How it ends →

From a number to a map

The first four articles established that a difference exists. This article asks where it lives. There is a change of kind here: every technique so far produced a number, and the techniques in this article produce a map. A number can be right or wrong. A map can be complete and correct and still point at the wrong thing, because attribution needs a model of which components exist before it can give cost to any of them.

The governing constraint is that time is spent by mechanisms that are invisible to the layer that measures them. The CPU cannot see that it is waiting for a disk. The query engine cannot see that its hash table missed the cache. The timer around an operator cannot see that the operating system paused its thread. Every instrument has a blind region, and the cost you are hunting is usually inside one.

Attribution is only as good as the inventory of things you were willing to attribute to.

Flamegraphs: on-CPU and off-CPU

A stack is the chain of function calls that are active at one moment: main called handle_request, which called write_hold, which called compress. A flamegraph is a picture built from many samples of the stack. The method is simple:

Take a sample of the call stack of the program at some frequency (for example, 99 times per second).
Group identical stacks.
Draw each function as a box. The width of a box is the fraction of samples that contained that function.

Two properties follow directly from this construction, and people misread them often.

Width is inclusive cost. A wide box means "time in this function and everything it called." The narrow boxes at the top are self time (time spent in the function's own code). A wide box near the bottom means "everything under here is expensive". It does not mean "this function is expensive". Optimizing the wide box itself is usually pointless: main is always the widest box and you cannot make main faster.

Sampling frequency sets a resolution floor. A profiler at 99 Hz takes a sample about every 10 ms. Anything shorter than 10 ms is visible only statistically, across many occurrences. A single rare 2 ms stall is structurally invisible: the sampler almost never lands inside it. This is the granularity-mismatch rule again (a mechanism that works in units of size X cannot see events much smaller than X). You can raise the rate, but each sample has a fixed cost, so a higher rate means more overhead. The overhead is the second old rule: a fixed cost per operation sets a natural rate. You trade blindness against perturbation (the profiler disturbing the program).

Which clock drives the sampling

The most important distinction is which clock decides when to sample.

On-CPU profiling samples only threads that are running on a CPU. A thread that is blocked for one second contributes nothing to an on-CPU profile. This is why the check from The right quantity comes first: compare wall-clock time with CPU time for the slow request. (If a request took 100 ms of wall-clock time and used 5 ms of CPU, the other 95 ms was waiting.) If the gap is large, an on-CPU flamegraph is a map of the wrong territory. It will be perfectly accurate about the 5% of the time that was not the problem.

Off-CPU profiling records blocking events instead. When the operating system takes a thread off the CPU, the profiler records the stack at that point and weights it by how long the thread stayed off the CPU. It finds lock waits, I/O waits, queue waits and scheduler starvation (a runnable thread that does not get a CPU). Its cost model is the reverse of on-CPU profiling: the number of events is proportional to the number of context switches (the OS pausing one thread and running another). A system that blocks a lot is expensive to profile. A system whose CPUs are saturated is cheap to profile.

Neither view is complete. On-CPU sees work, off-CPU sees waiting, and the sum is wall-clock time.

There is one more case to name. A system can show a flat on-CPU profile and a flat off-CPU profile and still be slow. That means the latency is outside the process: in the network, in a downstream service, or in the client. This is a finding, not a dead end. It tells you to stop profiling this process.