Replay
What makes a pipeline replayable, and what hazards come with replaying?
Replay makes a data platform recoverable, not just durable. It must be separated from its neighbors, because they are often mixed up.
| Scope | Trigger | Duration | |
|---|---|---|---|
| Retry | one record or batch | transient failure | seconds |
| Replay | a range of the stream | consumer bug, state corruption | minutes to hours |
| Reprocess | historical data through new code | logic change, new column | hours to days |
| Backfill | historical data into a new consumer | new downstream system | hours to days |
Retry is a flow-control mechanism. The other three are one mechanism, re-deriving output from retained input, and differ only in scope and intent.
The preconditions
A pipeline is replayable only if all five hold. Each is a design decision made earlier, usually without noticing.
- The input is retained long enough. Retention is not a storage-cost setting. It is your maximum recovery window. Someone who shortens it to save money has silently shortened your ability to fix bugs. (Retention is a contract between teams that has no interface.)
- Positions are addressable. You can say "start from here": offsets, snapshot IDs, logical dates, watermarks (markers of how far event time has progressed).
- Processing is deterministic: the same input gives the same output. It is broken by
now(), random sampling, iteration order over hash maps, floating-point addition giving different results under different parallelism, and lookups in a mutable dimension table (a reference table, such as seller details) that has changed since. - The sink is idempotent or supports atomic replacement. Replaying into an append-only sink duplicates everything.
- State can be reset or rebuilt. Replaying input into a consumer with stale accumulated state produces garbage. You must clear the state and replay from a point where the state was empty or checkpointed.
Item 3 is the one that hurts. now() inside a transformation makes your pipeline non-replayable, and nobody notices until a replay produces different numbers. The rule: a replayable pipeline is a pure function of its retained inputs. Anything that reads ambient, mutable or time-dependent context breaks the purity. Treat it as seriously as a correctness bug, because it is one, only deferred.
For dimension lookups there is a real design decision. Either snapshot the dimension and keep versions (so a replay uses the version that was live at that time), or accept that a replay gives current enrichment rather than historical enrichment. Both are defensible. Choose one and write it down, because the difference shows up as "the numbers changed" during your first recovery.
Replay hazards
- Replay storms. Replaying a month at full speed hits downstream systems at 100 times the normal rate. Pace the replay (Flow control) against a measured latency signal, not a guessed rate.
- Dedup window expiry. Your dedup store keeps 24 hours and you are replaying six weeks. Online dedup cannot help. You need replacement semantics (overwrite the partition, atomic swap) instead of incremental dedup. This is why replay usually needs idempotency by replacement, not idempotency by key.
- Ordering. Replayed records interleave with live records. If a downstream system applies "last writer wins" on arrival, a replayed old record overwrites current state. Version-based application fixes it: apply a record only if its version (or LSN, the position in the source's log) is higher than the stored one. This must exist before you need to replay.
- Side effects fire again. Replaying through a consumer that sends emails sends the emails again. Replay paths often need a different consumer configuration with external effects disabled. That switch must exist and must be tested.
- Downstream fan-out. Replaying one stage cascades into everything that reads its output. Replay is rarely a single-system operation. Know the blast radius before you start.
The design implication
The decisions that grant it are made months earlier, and each costs something at the time: log-based transport, enough retention, deterministic transforms, idempotent sinks, versioned dimensions, resettable state.
The argument that justifies the cost: replay is the only mechanism that fixes past wrongness. Everything else here prevents future wrongness. When you find a bug that has been corrupting output for three weeks, retries, backpressure and DLQs are all irrelevant. Only retention and determinism help. That is what you buy.