Four phases
The life of an incremental pipeline
The rest of this mechanism follows the life of an incremental pipeline in four phases.
| Phase | Question | What is in it |
|---|---|---|
| Detect | How do we find what changed? | log-based, trigger-based, query-diff, snapshot-diff, watermark-partition |
| Represent | What does a change event carry? | snapshot handoff, envelopes, op codes, before/after images, transaction metadata, ordering, schema changes |
| Apply | How does a change land on the target? | merge/upsert semantics, incremental view maintenance, feeding history tables |
| Repair | How do we fix and swap? | backfill and cutover, replay, dual-write avoidance |
Detect is an information-extraction problem with a fidelity ladder. Each rung down is cheaper and less coupled to the source, and loses a specific, nameable class of change. Polling is sampling: anything that changes faster than the poll interval is merged into a single observed transition, and anything that leaves no trace at all, such as a hard delete, is invisible in principle, not by accident.
Represent: a change event is only useful if it carries enough to be applied and enough to be applied idempotently and in order. The design question is what the event must carry so that a consumer who has lost its place can recover without asking the source.
Apply: here the question becomes algebraic. Which computations can absorb a change cheaply, and which cannot? This is where the area has real theory, and where marketing is furthest ahead of reality.
Repair: every incremental system needs a story for starting from nothing, for fixing itself after it is wrong, and for the moment when the old and new paths must swap without a gap and without an overlap.