Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Why restarting sometimes works

When does a restart clear an outage, and what is the correct recovery procedure?

Why "just restart it" sometimes works

A restart is a state-clearing operation. Whether it works depends entirely on where the sustaining state lives.

Restart clears in-process state: the request in local , heap and pressure, connection tables, accumulated locks and in-flight work. If the loop was sustained by any of these, a restart breaks it and the system returns to the healthy state.

Restart does not clear external state: client-side queues, upstream broker backlog, consumer lag, pending requests in load-balancer queues, and, most important, the clients' belief that they should retry. If the sustaining effect lives in the clients, a restart does nothing to it. It is waiting.

A restart makes the arithmetic worse on both sides

A restart arrives with C at its minimum: a cold cache (amplification 1/(1 − h), from cache stampede), empty connection pools, a not-yet-optimized JIT (the runtime's just-in-time compiler), an unwarmed buffer pool. And it arrives with λ at its maximum: the accumulated backlog plus a reconnect herd, all synchronized by the restart itself. You have deliberately created the worst possible point in the (λ, C) plane and invited the system to find an equilibrium there. The re-trigger is usually immediate.

The correct recovery procedure

To leave the broken state you must push λ below C/k, not below C. The two differ by the factor k. This is why "we shed 20% of traffic and nothing improved" is the expected result, not a puzzle. If k = 4, you needed to shed 75% (to reach λ below C/4, a 75% reduction from λ = C), and even that only works if C/k is computed with the cold-start capacity.

So the procedure is:

  1. Block traffic at the edge first.
  2. Drain or drop the backlog instead of serving it.
  3. Restart.
  4. Warm the caches.
  5. Readmit load progressively while watching for re-entry.

, which is politically hard and technically correct. Half-open probing in a circuit breaker is an automated search for exactly this recovery threshold.