Backfills, catchup and versioning

Why is catchup a design error?

CaseConstructed case · Tomás at Marigold Market

730 runs at once, and a torn partition

Tomás finds a bug in the cleaning step, fixes it, and wants to rebuild the last two years of daily_sales. He unpauses the workflow with a start date two years back. The orchestrator immediately launches 730 runs at once. The warehouse, sized for one run at a time, slows down; other teams' queries time out. Tomás pauses things and later runs the backfill carefully. A week later, a worker loses its network during a run. The orchestrator decides the task is dead and launches it again, but the first copy was alive. Both write to the same partition, and Nadia's report shows revenue for that day that matches neither copy of the data.

Three mechanisms are involved: backfill, idempotency and fencing.

How it ends →

Backfills, catchup and workflow versioning

Backfill is iteration over the interval-key space: run the workflow for each past interval. It is well defined only if the task is a pure function of its key. That is why the interval key (Graphs and time) and idempotency (below) are prerequisites, not companions. If the task reads now(), consults a mutable dimension table (a lookup table that changes over time), or appends instead of overwriting, then a backfill is not a replay. It is a different computation that happens to run on old dates.

Two decisions dominate.

Ordering

Sequential backfill preserves cumulative meaning: running balances, slowly changing dimension chains (history of changes of a record), and anything where interval N depends on N−1. Its wall-clock time is linear in the number of intervals.

Parallel backfill is bounded by capacity instead, but it is correct only when the intervals are genuinely independent. This is the trade of ordering versus parallelism with an unusually exact price tag: the cost of ordering is exactly the sequential backfill duration, and you can compute it before you decide.

Blast radius

A parallel backfill is a deliberate thundering herd (many clients hitting a resource together) aimed at a warehouse sized for one run at a time. The warehouse does not care that the load was intentional. A backfill needs its own concurrency bound and ideally its own resource pool, separate from production runs (concurrency controls). The same separation is needed to prevent starvation.

Catchup: a default that is a design error

Catchup means automatically running every missed interval when a paused workflow is resumed, or when a new workflow is deployed with a start date in the past. It deserves to be called out as a design error that became a default.

It turns an administrative action (unpausing) into an unbounded workload with no confirmation step.
The number of runs launched is a function of a date literal in a config file, which is a poor place for a capacity decision to live.
The defense is coherent: the schedule says those intervals exist, so they should run. But it is still the wrong default, because the failure is unbounded and the correct behavior is one keystroke away.

Versioning: the unsolved part

Versioning is the unsolved part, and it is better to say so than to offer a pattern. Pipeline code is mutable; run history is immutable. Backfilling 2024 with 2026's code produces a table whose rows carry two different definitions of the same column, with nothing to tell them apart. No orchestrator prevents or even detects this. The available answers are all unsatisfying:

Rebuild the whole table on every semantic change. Correct, but expensive and often impossible.
Stamp outputs with a code version and make consumers handle the mixture. This pushes the problem downstream, where it is worse.
Accept the silent mixture. Universal and undocumented.

Durable-execution systems face an acute version of the same problem: replaying an old event log against new control flow simply diverges. They respond with explicit versioning calls inside the code. That is the most honest treatment available, but it requires the developer to anticipate every future edit.

Backfill is only replay if the task is a pure function of its key; the code that replays it is not the code that wrote the history, and no orchestrator tracks that.