Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

The incident lifecycle

How does an incident turn into durable change, and why must state be captured before mitigation?

The incident lifecycle is the mechanism that converts an observation into durable change:

Mitigation and diagnosis have opposite information needs:

  • mitigation wants the fastest state change that stops the damage (restart, roll back, fail over; keeping a blue and a green copy of a table buys an instant roll back for 2× storage);
  • diagnosis wants the state preserved (the broken process, the logs, the memory).

So the order is not a matter of taste, and the conflict is real. The rule: (Take a heap dump, copy the logs, then restart.)

Blameless post-incident review has an informational justification, not a moral one. People withhold detail under threat of punishment. And the human account is the only record of internal state for the parts of the system you did not instrument. Blame degrades the sensor.