The USE method and per-operator profiles

How do you check every resource systematically, and what do per-operator profiles reveal?

The USE method, resource by resource

The USE method is a completeness discipline, not a tool. Its value is that it attacks the blind-inventory problem directly. Instead of starting from a symptom and following intuition, you first list the resources, and then check each one for three things.

Resources to list: CPU, memory capacity, memory bandwidth, each disk, each network link, and also the software resources that behave like resources: thread pools, connection pools, locks and memory grants. For each one record:

Utilization: the fraction of time the resource is busy.
Saturation: how much work is queued, waiting for it.
Errors: failures or rejections.

All three are needed because they do not move together in the way people expect:

High utilization, no saturation: a fully used resource that keeps up. That is efficiency, not a problem.
Moderate utilization with saturation: the arrivals are bursty, or the work is serialized. Work queues up even though the average looks safe. This is the variance argument from queueing theory in diagnostic form (queueing delay depends on burstiness as well as on average load), and it is the most commonly misread pattern in practice.
Errors often appear before saturation, as the earliest signal, because something upstream is already rejecting requests.

Saturation is the hardest of the three to get and the most valuable, because queue depths are often not exported by the software. When they are missing, you infer the queueing from the latency under a known arrival rate. This is Little's Law used backwards: L = λ × W.

The method's strength is that it finds resources you forgot to consider. Its weakness is that it stops at resources. It cannot diagnose a slow algorithm on an under-used machine, where everything is green and the code simply does too much work.

Per-operator profiles

This is a domain-specific instrument for query engines. It exists because generic tools attribute cost to the wrong axis. A flamegraph of a query engine shows time in "hash-table probe" and "decompression", added up across every operator that used them. Your real question is which operator (scan, filter, join, aggregate) was slow. Only the engine can answer, because only the engine knows how stack frames map to nodes of the query plan.

A per-operator view adds more than timing. Each item below has its own diagnostic value:

Rows in versus rows out per operator is the check of the optimizer's cardinality estimates (its predictions of how many rows each step will produce). Comparing estimated to actual rows is the fastest diagnosis of a bad plan. It points at the planner (the part that chooses the plan) and not at the execution engine (the part that runs it). A wrong plan is not a slow engine, and the remedies are unrelated.
Skew shows as the spread of time per task, or rows per partition, within one operator. The useful statistic is max over median. A skewed operator has a maximum that dominates while the median looks fine. So the total operator time is misleading, and the fix is to redistribute the work, not to tune the operator.
Spill makes the "systems are piecewise" boundary visible. The bytes spilled tell you the operator crossed from an in-memory algorithm into an external one that writes temporary data to disk. That is a step change in cost. A spilling operator's time cannot be compared with a non-spilling run of the same query.
Shuffle bytes and shuffle time separate computation from data movement (a shuffle is the all-to-all redistribution of rows between workers). They are two different costs with different remedies.

There is a systematic caveat: operator times overlap. In a pull-based iterator model (each operator asks its child for the next row), a parent's inclusive time contains the time of its children. So adding up operator times gives more than the wall-clock time, and the percentages do not mean what they appear to. Under true pipeline parallelism, operators run at the same time, and the critical path (the longest chain of dependent steps), not the sum, decides the duration. Reading a profile as a partition of total time is wrong in both cases.