Arrival exceeds service: four answers

When arrival rate exceeds service rate, what are the only four responses, and what does each sacrifice?

CaseConstructed case · Ravi at Cartwheel

Black Friday: the consumer killed by its own buffer

It is Black Friday. Orders arrive at Cartwheel at 2,000 per second, instead of the usual 300. The OrderPlaced events go into the broker without trouble. But the Inventory consumer can process only 800 events per second. The consumer keeps its incoming events in a list in memory. The list grows by about 1,200 events every second. After a few minutes the process runs out of memory, the operating system kills it (an out-of-memory kill, OOM), and it restarts. Every event that was in the list and not yet finished is gone. Stock counts are now wrong.

At the same moment a second thing happens. Cartwheel calls a partner API to validate delivery addresses. The partner starts answering 429 Too Many Requests and then 503. Cartwheel's retry code immediately calls again, so the partner receives even more calls. Soon every checkout waits 30 seconds for an address check.

Sana, the on-call engineer, sees no single broken component. Omar asks: "Can we just make the buffer bigger?" This article explains why that does not work, and what does.

How it ends →

Arrival rate exceeded service rate. Now what?

Transport patterns answered "how does a record cross the gap between two systems?" Flow control answers the question that matters once the system is running:

Two symbols appear often in this article. λ (lambda) is the arrival rate: how much work comes in per second. μ (mu) is the service rate: how much work the consumer can finish per second. In Ravi's case λ = 2,000 and μ = 800. When λ is larger than μ, work piles up somewhere.

Every mechanism in this article is one of four answers. The whole article collapses into this table:

Response	Mechanism class	What you sacrifice
Absorb the extra work	buffering, batching	latency, memory, freshness of data (staleness)
Slow the source	backpressure, rate limiting, throttling, pacing	the producer's freedom to send; risk of problems travelling upstream
Drop some work	load shedding, admission control	completeness
Sever the path	circuit breaker, bulkheading	availability of that path, on purpose

Two structural facts to hold before anything else.

First: absorbing is always temporary. A buffer (a waiting area for work) converts a rate problem into a time problem. If λ > μ for a long time, no buffer size saves you. The buffer only decides how long it takes until you must pick one of the other three answers. So every buffer needs a policy for what happens when it is full, and that policy is the real design decision. A buffer without a stated "what happens when full" rule is not a design. It is a crash that has been postponed.

Second: you do not get to skip the choice. If you do not choose, the system chooses for you. Memory grows without limit, the process is killed (OOM), and everything is dropped, including the in-flight work you would have liked to keep. The default choice is always the worst one. This is exactly what happened to Ravi's Inventory consumer.

This whole article is one trade-off seen from different sides: fail-fast versus fail-degraded.

Fail-degraded means the system keeps accepting everything and gets slower. Absorb and slow-the-source are of this kind.
Fail-fast means the system refuses some work early and stays healthy. Drop and sever are of this kind.

The mistake almost everyone makes early is to treat fail-degraded as the kinder option. But a degraded system that never recovers is worse than one that rejected 10% of the load and stayed healthy.