Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Load shedding

Which accepted work should be dropped, where, and how does a dropped request tell its client?

Load shedding means dropping work that you have already accepted, on purpose, to keep the rest healthy.

Why it is not a failure

Under overload, a system that tries to serve everything serves nothing well. Every request is slow, most time out, clients , and the useful output collapses to almost nothing while the system is 100% busy. Shedding 30% keeps the other 70% fast and stops the retry amplification.

is everything the system completes. is the part of it that is useful: requests answered in time for a client that is still waiting. Below saturation they are the same. Past saturation they diverge sharply.

The surprising fact: as offered load rises past capacity in an unprotected system, goodput does not stay flat. It falls. Wasted work (requests that will time out anyway, retries of those, abandoned work in progress) crowds out useful work. Shedding is what keeps the curve flat instead of falling.

What to drop: this is the whole design

Random dropping is the baseline and it is poor. These selectors are better, roughly in order of value:

  • By deadline. If the work will time out before you finish it, do not start it. This is a free win: you spend capacity only on work that can still be useful. It requires the deadline to be passed along with the request (a deadline header, an event expiry field). Build that early. It pays for itself again and again.
  • By criticality. Tag requests with a priority at the edge. Shed the batch , the recommendation call, the prefetch. Keep the checkout. It needs discipline across all services, but it is the most valuable classification you can add.
  • By cost. Drop expensive requests first, since one costly query may equal a hundred cheap ones. You need a cost estimate; a rough one is enough.
  • By tenant (fairness). Shed the noisy neighbor before the well-behaved one. This stops one tenant from consuming the shared shedding budget.
  • By staleness. For data that is replaced by newer data (metrics, prices, positions), drop the older values. You lose nothing that matters.
  • Sampling. For observability and analytics, drop statistically and record the sampling rate, so downstream users can correct for it. For aggregate this is almost free. Tail-based sampling means you decide after you have seen the outcome: keep all errors and all slow traces. It is strictly better than sampling (deciding at the start) when you can afford the buffer.

Where to drop

As early as possible. Work that was done and then thrown away is pure waste, and it was done while you were overloaded. Shedding at the edge (the load balancer, the API gateway, the ingestion service) is better than shedding at the database.

The failure to avoid is shedding after the expensive part. A request that is authenticated, calls three services and is then dropped at the last step consumed everything and produced nothing. Every shedding decision should be cheap compared with the work it prevents.

The signal

Do not shed based on request rate, because you do not know your capacity in requests. Shed based on a measured saturation signal:

  • Queue depth or queueing delay (CoDel-style: is there a ?).
  • Concurrency compared with a limit derived from : the number of requests in the system equals times time spent). Measure λ, choose a target W, and derive the limit L.
  • Adaptive concurrency limits, which find the limit from latency and need no configured capacity number: the modern default (concurrency limits).

The client contract

If you shed, say so in a way that can be told apart from a failure. Return 429 or 503 with a Retry-After header, and a clear meaning of "not attempted", so that retrying is safe. A shed that looks the same as a failure produces immediate, aggressive retries, which is the opposite of what you needed. Your own clients must also respect it. A retry budget (cap retries at about 10% of requests, across the whole fleet) is what stops the retry storm that shedding was supposed to prevent.

Admission control

drops work you have already accepted. declines it before any resources are committed, which is cheaper wherever it is available; shedding corrects the cases where admission guessed wrong. Its forms (concurrency limits, pool limits, reservation, priority and cost-based admission) are scheduling's.