Differential profiling

How does diffing against a matched control show the cost neither profile shows alone?

You have shown that A and B differ (Statistical hygiene). The differential profile tells you where.

Comparing two flamegraphs by eye fails, because both are dominated by the frames that are large in both. A differential flamegraph colors each frame by the change in its width. It surfaces the frames that moved, often narrow ones that you cannot see in either graph alone.

Three conditions make the diff valid. If you break any of them, you get confident nonsense:

Identical load. Both profiles must be taken under the same load. Otherwise the diff measures the load difference.
Enough samples. The change must exceed the sampling noise of each frame. This is the noise-floor requirement from Statistical hygiene, applied to each frame, and it is stricter than at the level of the whole total.
Comparable stack shapes. This breaks when inlining (the compiler copying a small function into its caller) changes between versions. The same work then appears under a different frame, and the diff shows a large decrease in one place and a large increase in another.

The idea worth carrying forward: differential methods are strictly more sensitive than absolute ones, at the price of needing a valid control. It is the same reason the paired, interleaved design of Statistical hygiene beats an unpaired comparison, and the same reason A/A tests set noise floors. Comparison against a matched baseline removes the variance that both sides share. The discipline lies in making sure the baseline is really matched.

What this article forbids

Using on-CPU profiling as the first step, without the wall-minus-CPU check.
Reading flamegraph width as self time.
Believing a frame that is supported by only a handful of samples.
Treating an instrumented binary as representative of the deployed one.
Concluding "CPU-bound" from high CPU utilization without looking at IPC.
Declaring a resource healthy on utilization alone.
Adding per-operator times and calling the sum a partition of wall-clock time.
Diffing profiles that were taken under different load.

Escapes and their prices

Escape	What it buys	What it costs
Sampling	Uniform, bounded overhead	A resolution floor: events shorter than the sampling interval effectively do not exist
Instrumentation	Exactness	Overhead proportional to event rate, uneven, which distorts the hot paths it measures
Adaptive or tail-triggered tracing	Visibility of rare events	A sample that is no longer representative
Hardware counters	The only view of why a cycle stalled	Multiplexing extrapolation, and often unavailable on shared hardware
Engine-level operator profiles	The correct attribution axis	Trust in the engine's own accounting, which is also instrumentation, with its own overhead and blind spots

One principle

Case, resolvedConstructed case · Ingrid at Bellwether Tickets

Slower at p99, and a flat profile

The corrected benchmark showed Tessera slower at p99 on the hold-write path, but its CPU profile was almost flat with cores at 35%, so nobody could say why. Where it started: Attribution and profiling →

Her on-CPU profile was flat and the cores were 35% busy. The first check from this article is wall-minus-CPU. For a typical slow hold-write, the wall-clock time is 9 ms and the CPU time is 1.2 ms. So 7.8 ms, about 87% of the request, is waiting. The on-CPU flamegraph was accurate about the 13% that was not the problem.

She switches to an off-CPU profile. One stack dominates: the commit path blocks inside a call that forces the write-ahead log to disk and waits for the disk to confirm (a synchronous flush, fsync). With a differential comparison against Engine A, the frame appears at once: Engine A batches the flush for many holds (group commit), while Tessera's default configuration flushes after every hold. She confirms with a second prediction, as a mechanistic explanation requires: if the flush is the cause, then Tessera's p99 should fall when the flush is grouped, and the disk's flush-wait counter should drop. Both happen.

She brings this to Chidi, the Tessera sales engineer. He is friendly and agrees at once: there is a group_commit setting, and he had a slide about it. Ingrid notes the lesson in her log: the result was not "Tessera is slower". It was "Tessera, with this default, waits on the disk." That is a different sentence, and Mateus can use it.