Recover from a stuck overload
Something briefly overloaded a system: a spike, a slow dependency, a deploy. The trigger is gone, yet the system stays down, because its degraded state sustains itself: retries, cold caches and backlogs keep it overloaded. Shedding a little traffic does nothing; restarting makes it worse. How do you get out?
You will end with how much load must be removed, and a recovery procedure that does not re-trigger the failure.
Work through it
Confirm it is stuck, not still triggered
If the original cause has gone and the system is still failing at normal traffic: the failure is sustaining itself (metastable). Find where the sustaining effect lives: inside the process (local queues, memory pressure, locks), or outside it (client retry queues, broker backlog, load-balancer queues, clients' belief that they should retry).
Work out how much load must go
The degraded system does k times more work per request. To leave the broken state, arrival rate must fall below capacity ÷ k, not below capacity. With k = 4, shedding 20% does nothing: you need to shed 75%. Compute capacity as it is at a cold start, which is lower than normal.
Understand why a plain restart fails
A restart clears state inside the process, so it works only if the loop lived there. It brings capacity back at its lowest (cold caches, empty connection pools, an unwarmed runtime) just as load arrives at its highest (the backlog plus every client reconnecting at once). It usually re-triggers the failure immediately.
Run the procedure
- Block traffic at the edge first.
- Drain or drop the backlog instead of serving it.
- Restart.
- Warm the caches.
- Readmit load progressively, watching for the system slipping back.
Every step deliberately destroys queued work, which is politically hard and technically correct.
Then lower k for next time
The width of the vulnerable range is set by k, not by spare capacity. Retry budgets, circuit breakers that probe recovery gradually, and warm-up ramps all lower it.