Glossary

93 terms, in the sense this mechanism uses them. The platform glossary has every term across all mechanisms.

A/A test: Comparing a configuration with itself to measure the noise floor.
A/B test: Comparing two versions under the same conditions to see if the difference is real.You compare two versions, A and B, under the same conditions and ask whether the difference is real. An A/A test compares the same version against itself, which shows how large the difference can be from noise alone.
Abort condition: A limit declared before a fault-injection or canary run; if reached, the experiment stops.
Aggregation: Replacing many observations with fewer numbers (a mean, a percentile). Lossy and irreversible.Combining many rows into fewer: SUM, COUNT, AVG. Aggregation loses detail. You cannot get the individual rows back from the total.
Arrival rate (λ): Number of new requests per second entering the system.
Benchmarketing: Consistent, one-directional choices in a published benchmark that favor the publisher.
Blast radius: The part of the system or the number of users affected when one failure happens.How much damage one mistake or one stolen secret can cause.
Bootstrap resampling: Drawing many new samples with replacement from your runs to estimate the spread of a statistic.
Cache: A deliberately kept second copy of data that no protocol forces to stay correct.1 other meaning →A cache is a small, fast place that holds copies of data kept somewhere bigger and slower. When a request finds its data in the cache, that is a hit. When it does not, it is a miss, and the system must go to the slower place (the backing store or origin). The hit rate h is the fraction of requests that hit. The miss penalty is the extra time a miss costs.
Canary: Sending a small fraction of real traffic to a candidate system to test it with the full loop.1 other meaning →
CDF: Cumulative distribution function: for each latency t, the fraction of requests finished within t.
Censoring: Hiding values beyond a limit (for example a histogram ceiling), usually making results look better.
ClickBench: A single-table analytic benchmark with no joins.
Closed-loop load: A fixed number of users who wait for each response before sending the next request.
Cold: A system whose caches, compiled code and connection pools are still empty, so it runs slower than when warm. A benchmark measured cold and one measured warm answer different questions.Many systems keep recently used data in fast memory (a cache). When the cache is empty the system is cold and slow. After it fills, the system is warm and fast. Other things also warm up: a JIT compiler (a runtime, as in Java, that compiles hot code while the program runs), connection pools, and so on.
Confidence interval (CI): A range that expresses the uncertainty of an estimate, such as 12% faster, 95% CI [7%, 17%].
Confounder: A second change, besides the one you test, that can explain the result.Something other than your change that also changed and could explain the result. If you test A on Monday and B on Tuesday, "the day" is a confounder.
Construct validity: Whether the quantity measured is the thing you actually care about.
Coordinated omission: A generator that waits for slow responses omits the requests it should have sent, hiding the worst latencies.
Decision rule (pre-registered): The criterion for acting, written down before the measurement.
Differential flamegraph: A flamegraph colored by the change in frame width between two profiles.
Effect size: How large the difference is, as opposed to whether it is statistically detectable.
Factorial design: An experiment that varies several factors together to find interactions.
Fan-out: Splitting work into many parallel tasks.1 other meaning →
Fault injection: Deliberately causing a failure to measure whether recovery mechanisms work.
Flamegraph: A histogram of sampled call stacks; frame width is the fraction of samples containing it.
Goodput: Completed work whose result is still wanted.Throughput is work completed per second. Goodput is work completed per second whose result is still wanted. If a client has already given up (timed out), finishing its request adds throughput but no goodput.
Gray failure: A partial failure that a component's own health check does not see but that clients experience.
Hardware counters: CPU registers that count events such as instructions, cache misses and branch mispredictions.
HDR histogram: A histogram with log-sized buckets giving constant relative precision over a wide range.
Heavy tail: A distribution in which a small fraction of items carry most of the total size.A distribution with a few extremely large values (a phone offline for a week) far above the typical ones.
Hedged request: Sending a duplicate request after a delay and using the first answer, to reduce the effect of random slow responses.
Held-out confirmation run: A fresh measurement of a selected winner, used to report its honest result.
Instrument: The tool that measures (load generator, profiler, metrics library).The tool that measures: a load generator, a profiler, a metrics library, a stopwatch. The system under test is the thing being measured.
Instrumentation: Adding code to record every event of a kind; cost grows with event rate.
Interference between arms: One arm of a live A/B test degrading the other through shared resources.
Interleaving: Alternating runs A, B, A, B so that environmental drift cancels in the comparison.1 other meaning →
Ipc: Instructions retired per CPU cycle; low values mean the core is mostly stalled.1 other meaning →
JIT compiler: A runtime that compiles hot code while the program runs.
Knee: The arrival rate where latency starts to rise sharply.1 other meaning →
Latency: Time one operation takes from request to response.Throughput is work finished per second. Latency is how long one operation takes. Little's Law links them to the number of operations in progress: L = λ × W, where λ is the arrival rate (per second), W is the average time one operation spends in the system, and L is how many are inside the system at the same time. Example: 1,000 requests per second, each taking 20 ms (0.02 s), means 1,000 × 0.02 = 20 requests in flight at any moment. This is an identity, not a design choice.
Little's Law: L = λW: average items in a system = arrival rate × average time in the system.Throughput is work finished per second. Latency is how long one operation takes. Little's Law links them to the number of operations in progress: L = λ × W, where λ is the arrival rate (per second), W is the average time one operation spends in the system, and L is how many are inside the system at the same time. Example: 1,000 requests per second, each taking 20 ms (0.02 s), means 1,000 × 0.02 = 20 requests in flight at any moment. This is an identity, not a design choice.
Macro-benchmark: A benchmark of a whole workload; real but hard to attribute.
Mean: The sum of values divided by their count.The mean is the average. The tail is the slowest few requests. "p99" means the value that 99% of requests are faster than, so 1% are slower. Averages hide the tail. A policy can improve the mean and make the tail worse.
Mergeable structure: A summary (like a histogram) that can be combined with others without losing the ability to compute percentiles.
Micro-benchmark: A benchmark that isolates one mechanism; clean but misses interference.
Multi-modal distribution: A distribution with several clusters, each often a different mechanism.
Noise: The part of a measured result that comes from chance rather than from the change being tested. Repeating runs shows how large it is; a confidence interval states it.Variance is how much results differ from each other when you repeat the same thing. Noise is the part of a result that comes from chance and not from your change. A confidence interval (CI) is a range such as "12% faster, 95% CI [7%, 17%]", meaning "the true improvement is probably between 7% and 17%."
Noise floor: The spread in the result when nothing has changed; found with an A/A test.
Observer effect: The change in the system caused by measuring it.
Off-CPU profile: A profile of where threads were blocked, weighted by blocking time.
Off-CPU time: Wall-clock time minus CPU time: time spent blocked or waiting.
On-CPU profile: A profile of where running threads spend time.
Open-loop load: Requests generated on a fixed schedule independent of responses.
Optional stopping: Running until a result looks significant; inflates false positives.
p99: The latency that 99% of requests beat; shows the slow tail.Time 1,000 requests and sort them from fastest to slowest. The p50 (median) is the 500th: half were faster. The p99 is the 990th: only 1% were slower. The p99.9 is the 999th. The slowest requests are called the tail. The mean (average) adds all the times and divides by the count.
Partly-open model: Independent arrivals where each user abandons after a timeout.
Per-operator profile: Timing, row counts, spill and shuffle statistics reported per plan node by the engine.
Percentile (p50, p99, p99.9): The value below which that fraction of observations fall.p50 is the median latency, p99 is the latency that 99% of requests beat. The slowest requests are the tail. When one request needs answers from many servers (a fan-out) it must wait for the slowest one. With 100 servers that each have a 1% chance of being slow, the chance that at least one is slow is 1 − 0.99¹⁰⁰ = 63%. So the user-facing tail is much worse than any single server's tail.
Practical significance: A difference large enough to matter for a decision.
Pre-registration: Writing the hypothesis and rules before seeing the data.
Query subsetting: Reporting only some benchmark queries, usually the favorable ones.
Queue: A line of work waiting for its turn, like people waiting at a bank counter. A scheduler is the part of a system that decides who is served next.1 other meaning →
Regression to the mean: Something selected for an extreme score tends to score closer to average when measured again.
Retry storm: Clients retrying failures add load that causes more failures.
Run-to-run variance: Spread of a summary statistic across repeated runs of the same configuration.
Sampling: Recording the state periodically; cannot see events shorter than the interval.1 other meaning →If you look at something only every 15 minutes, you see its state at those moments and nothing in between. If a value changes three times between two looks, you see only the last value. Signal engineers call this aliasing.
Saturation: Work queued waiting for a resource (part of the USE method).
Scale honesty: Stating the tested scale range and the mechanisms that would change outside it.
Scale ladder: Running the same test at increasing sizes (for example 1x, 3x, 10x, 30x).
Shadow traffic: Duplicating real requests to a candidate system and discarding its responses.
Sketch (HyperLogLog): A small summary of a data set that answers one kind of question approximately.A quantile is a percentile: the value below which a given fraction of data falls (the median is the 50% quantile). A sketch is a small, fixed-size summary of a huge dataset that answers one kind of question approximately: for example a t-digest or KLL sketch answers quantile questions, HyperLogLog counts distinct values, and Space-Saving finds the most frequent values. A sketch is built in a single pass and uses bounded memory.
SLO: Service level objective, for example "99% of requests under 100 ms".A service level objective is a target such as "99% of requests finish in under 100 ms."
Spill: Writing operator state to disk and continuing with a different algorithm, instead of failing.A hash join matches rows of two tables by building an in-memory hash table from the smaller side (the build side) and looking up each row of the other side. A sort orders rows. Both need memory proportional to their input. When there is not enough memory, a good engine writes some of its working data to temporary files on disk and reads it back later. This is called spilling. Part 5 explains it fully; Part 1 only names it.
Statistical significance: A difference unlikely to arise from noise alone.
Steady state: The period when the system's behaviour no longer drifts, used for measurement.
Steady-state hypothesis: A falsifiable statement of a metric staying inside a band during a fault experiment.
Synthetic data: Records sampled from a generative model fitted to real data.Fake data produced by a program for testing.
Tail (a log): The slowest requests, such as those above p99.p50 is the median latency, p99 is the latency that 99% of requests beat. The slowest requests are the tail. When one request needs answers from many servers (a fan-out) it must wait for the slowest one. With 100 servers that each have a 1% chance of being slow, the chance that at least one is slow is 1 − 0.99¹⁰⁰ = 63%. So the user-facing tail is much worse than any single server's tail.
Tail amplification: A fan-out's tail latency is worse than that of any single backend.
Think time: The pause a simulated user takes between requests.
Threats to validity: Specific ways an extrapolation can fail: confounding, hardware generation, workload survivorship, construct validity.
Throughput: Work completed per unit of time.Throughput is work finished per second. Latency is how long one operation takes. Little's Law links them to the number of operations in progress: L = λ × W, where λ is the arrival rate (per second), W is the average time one operation spends in the system, and L is how many are inside the system at the same time. Example: 1,000 requests per second, each taking 20 ms (0.02 s), means 1,000 × 0.02 = 20 requests in flight at any moment. This is an identity, not a design choice.
Throughput at SLO: The highest arrival rate at which the latency SLO is still met.
TPC-C / TPC-H / TPC-DS: Standard transactional (C) and analytic (H, DS) benchmarks with different structures.
USE method: Check utilization, saturation and errors for every resource.
USL: Scalability model with contention (α) and coherency (β) terms; predicts a throughput peak and then decline.
Utilization (ρ): The fraction of time a resource is busy.Utilization (ρ, "rho") is the fraction of time a server is busy. For random arrivals, average response time grows like S ÷ (1 − ρ), where S is the service time: 2×S at 50%, 5×S at 80%, 10×S at 90%, 100×S at 99%. Queues are caused by variability meeting load, and the curve has a wall at 100%. The last few percent of utilization cost more than the first eighty.
Variance: How much results differ from one another when the same thing is repeated. A difference smaller than the variance of repeated runs may be noise.Variance is how much results differ from each other when you repeat the same thing. Noise is the part of a result that comes from chance and not from your change. A confidence interval (CI) is a range such as "12% faster, 95% CI [7%, 17%]", meaning "the true improvement is probably between 7% and 17%."
Wall-clock time: Real elapsed time as the user experiences it.
Warm: A system whose caches, compiled code and connection pools are filled from earlier work, so it runs faster than when cold.Many systems keep recently used data in fast memory (a cache). When the cache is empty the system is cold and slow. After it fills, the system is warm and fast. Other things also warm up: a JIT compiler (a runtime, as in Java, that compiles hot code while the program runs), connection pools, and so on.
Within-run variance: Spread of individual operations inside a single run.
YCSB: A key-value benchmark with point reads and writes under configurable skew.