Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Write-ahead intent and the recovery scan

How does recording an intention before an effect make a crash resolvable, and what must the intent satisfy?

What it is. This is the general pattern underneath most of the others. A crash between two effects can be resolved afterwards only if the first effect recorded the intention to do the second. Recovery then becomes a scan for orphaned intents, each of which is either completed or reversed.

You have met this in many costumes: the WAL redo record, the transactional outbox (write the message to a table in the same database transaction as the data, and send it later), write-to-a-temp-file-then-rename, the _temporary directories of batch jobs, the lock record in Percolator (a transaction system built on a key-value store), saga logs (a record of the steps of a long business process, with compensating steps). One more you may not have counted: a reconciling control loop is a recovery scan that runs continuously instead of at start-up. The declared specification is the intent, and the controller (as in Kubernetes) sweeps for differences forever. Crash recovery and the whole operator pattern are the same mechanism at different duty cycles.

Example. Mediflow wants to "charge the card, then mark the result as billed". Before calling the provider, Wanjiru inserts a row intent(result_id, 'charge', state='pending') into her database. After the call she sets state='done'. After a crash, a sweeper reads all rows with state='pending' and completes or reverses each one.

Three requirements; the third is usually missed

  1. The intent must be durable before the effect.
  2. It must be idempotently completable: running the completion twice is harmless.
  3. It must be cheaply findable. If finding outstanding intents requires scanning all the data, recovery time scales with the size of the stored state instead of the size of the in-flight work.

The design rule to remember: Systems that violate it are the ones where start-up takes forty minutes and nobody can say why. The fix is always a small side structure: an active-transaction table, a "pending" directory, an indexed status column.

CREATE INDEX idx_intent_pending ON intent (created_at) WHERE state = 'pending';
-- the sweeper reads only this small index, not the whole table

Cost. One extra durable write on the critical path per operation, which means 2× writes at the granularity of one intent. (grouping the log writes of many operations into one disk flush) spreads that cost.

What it does not remove. The intent can be completed and the acknowledgment lost. The pattern removes internal ambiguity and pushes all that remains to one place: the reply to the caller. That is exactly where Making a repeat harmless's are installed.