Choose a recovery mechanism
A step in your system can fail part-way: a task dies, a machine is lost, a job crashes after writing half its output. Recovering it needs one of three things: a copy kept somewhere, a log of what happened, or a recipe to recompute it. Each puts its cost in a different place. Which mechanism fits this step?
You will end with a recovery mechanism for the step, where its cost lands, and the recovery time it delivers.
Decide
Is the step's effect visible outside the system and impossible to undo?
A card charged, an email sent, a parcel shipped.
YesCompensation
NoGo to 2
Why
Undo stops at the first effect someone outside has seen. No recovery mechanism can take it back; only a new action that offsets it.
Is the recovery-time target shorter than detecting the failure plus rebuilding the state?
A 30-second target, against a failure detector that waits 10 seconds and a restore that takes minutes.
Why
Every rebuild mechanism takes at least detection plus reconstruction time. Only a copy that is already running can beat that.
Is the result a pure function of inputs that are stored durably and never change?
Yes for a batch job over immutable files; no for a service whose state comes from requests that are gone.
Why
If the inputs and the computation are deterministic and still there, the recipe is enough and nothing extra needs storing. Otherwise you must keep the state itself, or a before-image to roll back to, and that cost lands on the write path.
Does recomputing from the start fit within the recovery-time target?
Where you can land
Compensation
Use it when the effect is external and irreversible.
A new action offsets the old one: a refund for a charge.
Before you build it: restructure so the one irreversible act comes last; then everything before it can still be recovered normally.
Standby replica
Use it when the target is shorter than detection plus any rebuild.
A warm or hot copy takes over.
What it costs: a full duplicate, always. Data written since the last replication is lost: your recovery point is the replication lag, measured at the worst moment, not on average. And the old primary must be fenced off at the shared resource, or two primaries write at once.
Checkpoints (or an undo log)
Use it when the state cannot be recomputed from durable inputs.
Before you build it
Set the interval
Checkpoint every √(2 × checkpoint stall × mean time between failures): with a 30-second stall and a failure a day, about 38 minutes. Count the stall (how long work pauses), not the volume written.
Recompute (lineage or replay)
Use it when the result is a pure function of durable inputs and recomputing fits the target.
Store nothing extra; rerun the recipe from the inputs.
Before you build it: check that the computation really is deterministic: now(), random numbers and lookups into changing tables break it, and nothing detects the difference. Replay must also drain faster than new data arrives, or it never finishes.
Recompute, with checkpoints to cut the distance
Use it when the result can be recomputed, but not within the target.
Before you build it
Place the checkpoints
For work that runs over time: at √(2 × stall × mean time between failures). For a graph of stages: at the shuffle boundaries, where one lost output would otherwise mean recomputing everything that fed it.
Whichever you land on
Assume the redo lands twice
Every mechanism here can redo work that had partly happened. Make the repeat harmless: see Make a repeat harmless.