Rate limiting, throttling and pacing

Why does pacing need jitter?

Rate limiting

Rate limiting is a policy applied at a boundary. You have decided in advance the maximum rate you will accept. It is not a reaction to load. It is a contract declared beforehand. This is what separates it from backpressure (below), which is a measured signal from the actual consumer.

The algorithms and what actually differs

Token bucket. Imagine a bucket that receives tokens at a rate R per second, up to a capacity C. Each request takes one token. If the bucket is empty, the request is rejected or has to wait.

It allows bursts up to C, then settles to rate R. This is a feature, not a defect. Real traffic comes in bursts, and a limiter that forbids bursts rejects legitimate work.
It needs O(1) state per key: the time of the last refill and the token count. It is easy to share between machines using one shared counter with a CAS (compare-and-swap, an atomic "change this value only if it still has the value I last saw").
C is the interesting setting, and the one people leave at a default. C is how much of a burst you are willing to take. C = R means no burst tolerance at all.

Leaky bucket. Requests enter a queue that drains at exactly rate R.

The output is perfectly smooth. Nothing downstream ever sees a burst.
Bursts are absorbed into the queue (adding latency) or rejected when the queue is full. They never pass through.
Use it when the thing you protect is rate-fragile, not volume-fragile: a downstream that can handle 1,000/s all day but collapses at 3,000/s for a single second. It is also the right shape when you are a polite user of someone else's API.

The one-line difference: a token bucket limits the average and permits bursts; a leaky bucket enforces the shape.

Fixed window counter. Count requests per clock window (for example per minute) and reset at the boundary. It is simple and broken at the boundaries: R requests at 0:59 and R more at 1:01 means 2R in two seconds. It is acceptable only when 2R is survivable.

Two bursts across a window boundaryillustrative · limit 100 per minute

Sliding window log. Keep the timestamp of every request in the window and count those still inside. It is exact, and uses O(number of requests) memory per key. It is correct but does not scale to many keys.

Sliding window counter. The practical compromise. Interpolate between the previous and the current fixed window according to how much of the current window has elapsed. It uses O(1) memory, is approximately correct, and has no boundary spike. This is a trade of exactness for approximation, and it is almost always the right trade: nobody needs a rate limit to be exact to the single request.

Distributed rate limiting: the part that is actually hard

On one machine this is easy. Across N machines there are three approaches:

Local limits of R/N each. No coordination, but wrong under skew. A hot key that lands on one machine is throttled at R/N while the rest of the fleet sits idle.
A shared store (for example a Redis counter with an atomic increment). It is accurate, but it adds a network round trip to every request and a hard dependency on the store. Decide the behavior when the store is unavailable, and write it down. Fail-open means the limiter disappears exactly when you are under load. Fail-closed means you have made the limiter a single point of failure for the whole service. Both are bad. Choose knowingly.
Local token buckets with periodic global reconciliation. Each machine holds a local allowance, borrows more from a central authority when needed, and returns what it did not use. It is approximately correct, needs no round trip per request, and degrades gracefully. This is where production systems end up, and it is the answer worth knowing.

Throttling versus rate limiting

These two words are often used as if they were the same. They are different.

Rate limiting is a hard cap at a boundary. It is binary: allowed or refused.
Throttling is continuous, graduated slowing under load. It is not a cap but a response function: increasing artificial delay, reduced concurrency, a degraded service tier, longer polling intervals.

The graduated shape has a real advantage. It turns a cliff into a slope. A rate limiter goes from 100% service to hard rejection at one request. A throttle degrades smoothly, which gives clients time to react and gives you a warning signal before you reach the wall. Where you have a choice, prefer the slope.

Pacing and smoothing

Rate limiting says "no more than R". Pacing says "spread the R you send evenly." The goal is different: it is not about the average, it is about the instantaneous shape of the traffic.

It matters more than it looks:

Synchronized bursts. A cron job scheduled as 0 * * * * on 200 machines means 200 simultaneous requests at the start of every hour. The average load is small. The instantaneous load is 200 times larger. The fix is to add jitter (a small random delay) to the schedule. This is the best value-per-line-of-code fix in this whole article.
Retry storms. If every client uses the same backoff, the herd stays together and fails together. Jitter is not optional in backoff. Why, and why full jitter wins, is in Retry with exponential backoff and jitter.
Downstream buffer protection. A burst that is inside your rate limit can still overflow a downstream buffer that was sized for the average. Pacing prevents this without any coordination.
Backfill etiquette. Replaying six months of history at full speed will destroy a downstream that is also serving live traffic. Pace the backfill to a fixed fraction of capacity. Better still, make the fraction depend on measured live latency, not on a fixed rate you guessed.

Mechanically, pacing is a leaky bucket applied to your own output, or a scheduled release with jitter. It is cheap, and it prevents a type of incident that is very hard to diagnose afterwards, because all the average metrics look fine.