Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Backlog burst and replay storms

How long does a backlog take to drain, and why does catching up overload the next stage?

A B drains at rate (C − λ), so an outage of duration T at ρ takes T × ρ/(1−ρ) to clear: a one-hour outage on a comfortable-looking 90%-utilized pipeline takes nine hours (the catch-up condition). That is the optimistic case: it assumes catch-up runs at full capacity and nothing downstream objects.

The second-order effect

Catch-up runs at C, not at λ, but everything downstream was sized for λ. So recovering one stage overloads the next. If the next stage has an amplification factor above one, it falls into its own broken equilibrium. This is a cascade that spreads through time rather than across the dependency graph. The recovery of a system is a load event that no capacity plan includes: the peak load event in a system's life is its own recovery, not its traffic. Recovery combines backlog drain, a synchronized reconnect herd, cold capacity, and rebuild or rebalance traffic, all at once and all correlated by the incident. Predictive form: for any system, compute the recovery load explicitly and check whether it exceeds the provisioned capacity. If it does, then recovery is not a procedure you have. The real fix is usually to make part of the backlog droppable, not to buy capacity for an event that happens once a year.

Escapes

All escapes refuse to recover at full speed:

  • Rate-limit the drain to protect downstream. You pay in staleness.
  • Reserve burst capacity that is used only for catch-up. You pay in idle resource.
  • Newest first, backfill history separately ( again), because during catch-up only the freshest events matter to anyone.
  • Drop the backlog when the data's value decays faster than the recovery time: metrics, cache refreshes, presence updates, recommendations. Ask what the backlog is worth when it finally arrives. A surprising amount is worth nothing.