Glossary

116 terms, in the sense this mechanism uses them. The platform glossary has every term across all mechanisms.

a^N (availability ceiling): Availability of a service with N hard dependencies, each available with probability a; for example 0.999^30 = about 97%.
Admission control: Declining new work before any resources are committed to it.
Aging: Raising a waiting task's priority over time to prevent starvation.
AIMD: Additive increase, multiplicative decrease: a rate-control rule that grows slowly and cuts sharply when congestion appears.
Availability: Availability is the fraction of time a service works. 99.9% is "three nines", 99.99% is "four nines". Over a year of 8,760 hours, 99.9% allows about 8.76 hours of downtime and 99.99% allows about 52.6 minutes.1 other meaning →
Backlog: Work that accumulated while a system could not keep up, waiting to be processed.
Backpressure: A signal from a consumer, based on its actual state, that makes the producer slow down.If a slow step cannot keep up, it tells the step before it to slow down, and that step tells the one before it, until the source stops reading. We study this in Part 8.
Bimodality: A system that is stable in two very different states (healthy and degraded) with nothing stable in between.
Bistability / metastable: A system with two stable states; a metastable failure stays in the bad one after the trigger is gone.
Blast radius: The part of the system or the number of users affected when one failure happens.How much damage one mistake or one stolen secret can cause.
Bounded queue: A queue with a maximum length that rejects new work when full, turning waiting into a fast error.
Branching process (R): A model where each request causes R new requests on average; R < 1 dies out, R > 1 grows.
Broadcast join: A join that copies the small side to every executor so the large side does not move.
Brownout: Serving all users a reduced response (for example without recommendations) instead of refusing some users.
Bufferbloat: Excess latency caused by large buffers that stay full, with no gain in throughput.
Buffering: Storing excess work inside the system; latency and memory grow without bound.Latency is how long one operation takes. Throughput is how much work finishes per second. Buffering means putting a queue or a temporary store between a fast side and a slow side, so neither one waits for the other.
Bulkhead: A partition of resources so a failure in one part cannot consume the resources of another.
Cache stampede: Many requests miss the same expired or missing cache key and all go to the backend together.
Capacity: How much work a system can finish per second (C, or μ for one server). Utilization ρ = arrival rate ÷ capacity is the share of it in use.Load is how much work arrives per second, written λ (lambda, "arrival rate"). Capacity is how much work the system can finish per second, written C (or μ, "mu", for one server). Utilization is the fraction of capacity in use, ρ = λ / C (rho). At ρ = 0.5 the system is half busy.
Cell: An independent full copy of a service serving a subset of customers to limit blast radius.
Circuit breaker: A mechanism that stops calls to a failing dependency (closed, open, half-open).
Coffman conditions: The four conditions for deadlock: mutual exclusion, hold-and-wait, no preemption, circular wait.
Compaction: Rewriting many small files into fewer larger ones.An LSM-tree (Log-Structured Merge tree) is a storage structure that never changes files in place. It collects writes in memory, writes them as sorted, immutable files, and later compacts (merges and rewrites) the files in the background to remove old versions and deleted entries. A tombstone is a marker that says "this key was deleted"; compaction removes the key and the marker. Compaction costs disk and CPU. Write amplification is the ratio of bytes the storage actually writes to bytes the application asked to write. A Bloom filter is a small structure that answers "definitely not in this file" or "maybe in this file", so reads can skip most files.
Congestion collapse: A state where throughput stays high while goodput falls because work is wasted.
Control: The data plane is the part of a system that handles user requests (route this packet, answer this query). The control plane is the part that manages the system (add servers, change configuration, update routing tables, decide who is healthy).
Control plane vs data plane: The control plane manages the system (autoscaler, config, discovery); the data plane serves requests.
Convoy effect: Threads moving in lockstep behind a lock after its holder was descheduled; throughput falls to context-switch speed.
Critical section: A piece of code protected by a lock so that one thread at a time runs it.
Data plane: Where the real data is stored and moved.The data plane is where the real data moves (storage, Spark, the warehouse). The control plane is the layer that only sends commands and receives status. The orchestrator is a control plane.
Dead time: Total delay between a real state change and the effect of a controller's response to it.
Dead-letter queue (DLQ): A holding area for records rejected by validation.A place where a pipeline puts records it could not process or did not accept, so they are kept instead of lost.
Deadline propagation: Passing the remaining time budget of a request down the call chain so each hop can drop expired work.
Deadlock: A group of waiters in a cycle, each holding what another needs, so none progress.
Debt window: The gap between peak write absorption (buffer size) and sustained throughput (bandwidth over write amplification).
Decorrelated jitter: A backoff method where each delay is random and based on the previous delay, so clients do not stay synchronized.
Deferred work: Work postponed to a later phase (collection, compaction, catch-up) that must use the same resource as foreground work.
Delay (in a loop): The time between a change in system state and the reaction to it; long delay makes control unstable.
Differential observability: Different observers (health checker, client, peers) see different health for the same component.
Discard at dequeue: Checking a request's deadline when it leaves the queue and dropping it at no cost if it has expired.
Efficiency opposes independence: Sharing, standardizing and automating make components fail together.
Eviction cap (fail open): A limit on how many nodes a health checker may remove; beyond it, nobody is removed and a human is paged.
Fail-slow: A component that keeps working but very slowly; harder to detect than one that stops.
Fail-stop: A component that either works correctly or stops and is clearly dead.
Failure detector: A mechanism that decides whether a component has failed, usually from timeouts; it cannot be perfectly correct in an asynchronous system.Any piece of software that decides "that machine is dead". Since a machine cannot announce its own death, every detector works by waiting for a heartbeat or reply that does not come. So every detector is a timeout in disguise.
Feedback: When a process's output influences its own input. Negative feedback opposes change and stabilises (a thermostat); positive feedback reinforces it and runs away (a microphone near its own loudspeaker).A feedback loop exists when the output of a process influences its own input. A thermostat is the classic example: the room temperature turns the heater on or off, and the heater changes the room temperature. In negative feedback the correction opposes the change (the thermostat). In positive feedback the change reinforces itself (a microphone near its own loudspeaker squeals louder and louder).
Fencing: A mechanism (such as a token number) that makes a stale or old leader unable to write after a failover.
FIFO / LIFO: First in, first out and last in, first out queue orders.
Gain: The ratio of extra output to input in a loop; how many extra requests each request causes.
Garbage collection (GC): Runtime component that finds unreachable objects and reclaims their memory.In Java, Go, C# and similar languages you never call free. A background component, the garbage collector, finds objects that nothing refers to any more and reclaims their memory. A tracing collector does this by starting from known roots (variables) and following every reference to mark the objects that are still live. The live set is the total size of all live objects. A generational collector splits the heap into a young generation (new objects, collected often and cheaply) and an old generation (objects that survived several collections). Pauses are moments when the program is stopped so the collector can work.
Generational GC: A collector that treats young objects separately, relying on most objects dying young.
Goodput: Completed work whose result is still wanted.Throughput is work completed per second. Goodput is work completed per second whose result is still wanted. If a client has already given up (timed out), finishing its request adds throughput but no goodput.
Goodput vs throughput: Throughput counts all work done; goodput counts only work that someone still wanted.
Gray failure: A partial failure that a component's own health check does not see but that clients experience.
Half-open state: Breaker state where a few probe requests test whether the callee recovered.
Hard vs soft dependency: A hard dependency must work for the request to succeed; a soft one has a fallback or can be skipped.
Head-of-line blocking: An item at the front of a queue blocking all items behind it.
Headroom: Spare capacity held to absorb growth during the autoscaling loop delay (ramp rate × delay).
Hedged request: Sending a duplicate request after a delay and using the first answer, to reduce the effect of random slow responses.
Hot key: A key with far more rows than average.One key that receives far more traffic than others (a famous artist's tickets, one celebrity's profile). Traffic to keys is often Zipf-distributed: a few keys take most of the load.
Hysteresis: Behavior that depends on history: the threshold to enter a state differs from the threshold to leave it.A system whose behavior depends on its recent past, so it does not flip back and forth at a single threshold. A thermostat that turns on at 19 °C and off at 21 °C has a 2-degree hysteresis band. It avoids flapping, at the price of a wider swing.
Independence: Two events are independent if one happening tells you nothing about the other. Most reliability arithmetic assumes that failures are independent. This mechanism shows how often that assumption is false.
Jitter: Randomness added to waiting times to avoid synchronized clients.
k (degraded-mode amplification): Work done per unit of useful demand when the system is in the degraded state.
Latency: Time one operation takes from request to response.Throughput is work finished per second. Latency is how long one operation takes. Little's Law links them to the number of operations in progress: L = λ × W, where λ is the arrival rate (per second), W is the average time one operation spends in the system, and L is how many are inside the system at the same time. Example: 1,000 requests per second, each taking 20 ms (0.02 s), means 1,000 × 0.02 = 20 requests in flight at any moment. This is an identity, not a design choice.
Leverage 1/(1-h): How strongly a cache hit rate h protects the backend: miss traffic is 1 - h, so a drop in h multiplies backend load.
Little's Law: L = λW: average items in a system = arrival rate × average time in the system.Throughput is work finished per second. Latency is how long one operation takes. Little's Law links them to the number of operations in progress: L = λ × W, where λ is the arrival rate (per second), W is the average time one operation spends in the system, and L is how many are inside the system at the same time. Example: 1,000 requests per second, each taking 20 ms (0.02 s), means 1,000 × 0.02 = 20 requests in flight at any moment. This is an identity, not a design choice.
Live set: Total size of the objects that are still reachable; determines tracing GC cost.
Livelock: Threads constantly change state in response to each other and make no progress.
Load: How much work arrives per second, written λ (the arrival rate). Divided by capacity it gives utilization ρ, the share of capacity in use.Load is how much work arrives per second, written λ (lambda, "arrival rate"). Capacity is how much work the system can finish per second, written C (or μ, "mu", for one server). Utilization is the fraction of capacity in use, ρ = λ / C (rho). At ρ = 0.5 the system is half busy.
Load shedding: Deliberately dropping work already accepted to keep the rest healthy.
Lock-free structure: A data structure that lets threads make progress without holding a lock.
Loop: A chain in which a system's output affects its own input.
LSM-tree: Log-structured merge tree: buffer writes in memory, flush sorted runs, merge in the background.An LSM-tree (Log-Structured Merge tree) is a storage structure that never changes files in place. It collects writes in memory, writes them as sorted, immutable files, and later compacts (merges and rewrites) the files in the background to remove old versions and deleted entries. A tombstone is a marker that says "this key was deleted"; compaction removes the key and the marker. Compaction costs disk and CPU. Write amplification is the ratio of bytes the storage actually writes to bytes the application asked to write. A Bloom filter is a small structure that answers "definitely not in this file" or "maybe in this file", so reads can skip most files.
Masking: Hiding a failure by retries, correction or fallback; it moves the failure into a channel with no counter.1 other meaning →
Memtable: The in-memory write buffer of an LSM-tree.
Multiplexing: Carrying many independent streams over one connection.1 other meaning →
Off-heap memory: Memory outside the managed heap, invisible to the garbage collector, managed explicitly.
Peer-relative comparison: Judging a node by comparing it with the others in a homogeneous group instead of with a fixed threshold.
Percentile (p50, p99, p99.9): The value below which that fraction of observations fall.p50 is the median latency, p99 is the latency that 99% of requests beat. The slowest requests are the tail. When one request needs answers from many servers (a fan-out) it must wait for the slowest one. With 100 servers that each have a 1% chance of being slow, the chance that at least one is slow is 1 − 0.99¹⁰⁰ = 63%. So the user-facing tail is much worse than any single server's tail.
phi-accrual detector: A failure detector that outputs a suspicion level from observed heartbeat timing instead of a yes or no.
Poison pill: A record that fails deterministically on every retry and can block an ordered partition.
Poisson arrivals: A model of independent random arrivals at a constant average rate.
Power-of-two-choices: Pick two servers at random and send to the less loaded; avoids herding.
Premature promotion: Moving objects to the old generation too early because their request was still in flight.
Priority ceiling: A lock carries the highest priority of any task that may take it, and the holder runs at that priority.
Priority inheritance: A lock holder temporarily takes the priority of the highest-priority waiter.
Priority inversion: A high-priority task effectively runs at lower priority because it waits for a resource held by a lower one.
Queue: A line of work waiting for its turn, like people waiting at a bank counter. A scheduler is the part of a system that decides who is served next.1 other meaning →
QUIC: A transport protocol that makes streams independent, used by HTTP/3.
Repellor: A state the system moves away from, so it does not stay there (an unstable point between two stable states).
Replay storm: Overload caused by reprocessing a backlog at full speed after an outage.
Retry: If an ack does not arrive within some time, the sender gives up waiting (a timeout) and often sends again (a retry). Retrying is useful, but it can create duplicates, as you will see.
Retry amplification: The multiplication of load by retries: (1 + r)^d for r retries over d tiers.
Retry budget: A limit that allows retries only up to a fraction of recent successful requests.
RUM trade-off: Read, update and memory amplification: improving one worsens another.
Salting: Adding a random suffix to a hot key so it splits into k sub-keys.
Shared fate: Components that fail together because they share a resource, code, configuration or deploy path.
Shuffle sharding: Assigning each tenant a random subset of k nodes out of n so that few tenants share the same subset.
Slow-start / progressive readmission: Raising admitted traffic in steps as a restarted component warms up.
Starvation: One class of work is never served while the system is otherwise busy.
Static stability: The ability of the data plane to keep working with last known state when the control plane fails.
Straggler: A task much slower than its peers.
Tail (a log): The slowest requests, such as those above p99.p50 is the median latency, p99 is the latency that 99% of requests beat. The slowest requests are the tail. When one request needs answers from many servers (a fan-out) it must wait for the slowest one. With 100 servers that each have a 1% chance of being slow, the chance that at least one is slow is 1 − 0.99¹⁰⁰ = 63%. So the user-facing tail is much worse than any single server's tail.
The queueing wall: The sharp rise of response time as utilization approaches 1: R ≈ S / (1 − ρ) gives 2S at ρ = 0.5, 10S at 0.9 and 100S at 0.99. It grows slowly, then very fast.A single server with random arrivals has average response time roughly R = S / (1 − ρ), where S is the service time with no waiting. At ρ = 0.5, R = 2S. At ρ = 0.9, R = 10S. At ρ = 0.99, R = 100S. Response time does not grow in a straight line. It grows slowly and then very fast as ρ approaches 1. This is the queueing nonlinearity.
Throughput: Work completed per unit of time.Throughput is work finished per second. Latency is how long one operation takes. Little's Law links them to the number of operations in progress: L = λ × W, where λ is the arrival rate (per second), W is the average time one operation spends in the system, and L is how many are inside the system at the same time. Example: 1,000 requests per second, each taking 20 ms (0.02 s), means 1,000 × 0.02 = 20 requests in flight at any moment. This is an identity, not a design choice.
Thundering herd: Many clients acting at the same time, such as retrying or refreshing together.
Tied request: Requests enqueued at two replicas, each cancelling the other when it starts.
Timeout: The maximum time a caller waits before giving up.A timeout is the longest a caller waits for a reply before giving up. A retry is sending the same request again after a failure or timeout. Retries help when the failure was a rare accident. They add load when the failure is caused by overload.
Token bucket: A limiter that adds tokens at rate R up to capacity C; allows bursts up to C.
Trigger vs sustaining effect: The trigger starts an incident; the sustaining effect keeps it going after the trigger is gone.
USL beta (retrograde): The coherency term of the Universal Scalability Law; when it is positive, throughput falls past a peak load.
Utilization (ρ): The fraction of time a resource is busy.Utilization (ρ, "rho") is the fraction of time a server is busy. For random arrivals, average response time grows like S ÷ (1 − ρ), where S is the service time: 2×S at 50%, 5×S at 80%, 10×S at 90%, 100×S at 99%. Queues are caused by variability meeting load, and the curve has a wall at 100%. The last few percent of utilization cost more than the first eighty.
Warm pool: Pre-started spare instances that reduce provisioning delay.
Write amplification: Bytes written per byte of user data.If you write 1 MB of data and the system physically writes 10 MB (because of rewriting and merging), the write amplification is 10.
Write stall: A deliberate halt of writes when compaction debt crosses a threshold.
Zipf distribution: A distribution where the r-th most common value has frequency about 1/r of the top one.In many workloads a few items are requested very often and most items are requested rarely. Zipf's law says the item of popularity rank i (rank 1 is the most popular) is requested with frequency proportional to i^(−α). The number α (alpha) is the popularity exponent. A larger α means a steeper concentration on the top items.