Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Runbooks and auto-remediation

When does a runbook become code, and why must auto-remediation itself be watched?

A is a written procedure for a known problem ("if the alert fires, check the source feed, then restart job X"). It is a cached decision procedure: you pay the thinking cost once, calmly, and reuse it under stress. Its value is frequency × cognitive load under stress. The stress term is why the cache is worth maintaining even for steps the author finds obvious: at 03:00, nothing is obvious.

A progression follows:

  1. If a runbook is fully deterministic, it is code.
  2. If it is code, it is (the system fixes itself).
  3. Auto-remediation is a control loop, and an ungoverned control loop can sustain a failure.

Example: automatic restart on out-of-memory (OOM). It removes the symptom, hides the leak, and can hold the system in a stable-broken mode forever: it never fully fails and never gets fixed.

The required guards have the same shape as retry budgets:

  • rate-limit the remediation (at most N restarts per hour);
  • log every invocation;
  • escalate on repeat within a time window.