Catching up after an outage

How it behaves

One input on the horizontal axis, the others held fixed.

Worked examples, recomputed

Every number the mechanisms quote for this relationship is recomputed from it, so none of them disagrees with it.

• 3 h outage at 50% utilization → another 3 h to catch up
• recover in a third of the outage → run at 25% utilization
• at 90% utilization the drain is 1/9 of arrivals: 9× the outage
• ρ = 0.5 → 1 × the outage
• ρ = 0.8 → 4 × the outage
• ρ = 0.9 → 9 × the outage
• a 10-minute outage at ρ = 0.8 → 40 minutes to drain
• 10-minute outage at 90% → 5,400 s
• stop 6 h at 60% → 9 h to catch up
• 600,000 records of lag, 5,000 − 4,000 per second → 600 s

Where it appears

Each mechanism states it in its own names. The general symbol each name stands for is shown with it.

• Fault Tolerance & Delivery Semantics

  What is the cheapest thing that can be safely done twice?

  $$\text{recovery time}=\text{backlog}/(\text{replay throughput}-\text{arrival rate})$$

  Read it in place →

• Emergent Failure Dynamics

  What happens when nothing is broken, every component is healthy, and the system's own response to stress makes the stress worse?

  $$T_{\text{recovery}}=\frac{\lambda T}{C-\lambda }=T\cdot \frac{\rho }{1-\rho }$$

  C = $M$ · λ = $a$

  Read it in place →

• Streaming

  What does it cost to remember, when the input never ends?

  $$\frac{\text{arrival}\times \text{outage}}{\text{replay}-\text{arrival}}$$

  Read it in place →

• Observability & Operations

  What is the smallest description of a running system from which we can find a fault, fast enough to matter, at a cost we can still pay while the system is failing?

  $$\text{time to drain}=\frac{\text{lag}}{\text{consume rate}-\text{produce rate}}$$

  Read it in place →