Summary and key takeaways
The reframe. CDC extracts a derivative from a system that treats its own history as private. It introduces two constraints that earlier topics lacked: ownership (the source belongs to someone else) and path dependence (an incremental result is a fold over a history, correct only if nothing is lost, duplicated or reordered).
The break-even model. Full recompute costs S·c. Incremental costs Δ·S·c′ + F. Incremental wins when Δ < (1 − F/(S·c)) · c/c′. With a 20× merge penalty the crossover is near 5%. Below some absolute size, F makes incremental strictly worse.
The four phases:
| Phase | Key points |
|---|---|
| Detect | Fidelity ladder: log-based (retention coupling, inverted blast radius), trigger-based (2–5× write latency), polling (aliasing, invisible hard deletes, unreliable watermark column). Every method has a bypass set it cannot measure. |
| Represent | Envelope = op + before/after images + source position + transaction id + two timestamps. Signed multiplicity (update = −old, +new). Key-partitioned transport drops transaction atomicity. Schema is an in-band ordered fact. |
| Apply | Merge needs a unique, stable, non-null key. Cost scales with substrate granularity (partition, copy-on-write file, merge-on-read row). IVM: cheap only with an inverse (SUM, COUNT yes; MAX, quantiles no); joins need full state. |
| Repair | Overlap deliberately, idempotent sink. Replay horizon = retention. Never dual-write; use CDC or outbox. Commit cursor with state. Type 2 history needs path-complete capture. |
Coarse methods that avoid CDC: watermark-based incremental batch (append-only, partitioned by arrival; add a re-processing window) and snapshot-diff on versioned tables (the cursor races the garbage collector).
Equations and formulas:
| Formula | Meaning |
|---|---|
| Δ < (1 − F/(S·c)) · c/c′ | break-even change fraction |
| Δ(R ⋈ S) = ΔR⋈S + R⋈ΔS + ΔR⋈ΔS | incremental join, needs both operands as state |
| update = (old, −1) + (new, +1) | signed multiplicity |
| new_sum = old_sum − deleted | why a group (inverse) makes deletes cheap |
Four principles earned here:
- Observability of change is a property of the source.
- Incrementality requires an inverse.
- Derived state drifts; only recomputation proves otherwise.
- A private detail, once consumed, becomes a public interface.
One repeating shape: retention is a negotiation between a collector and the slowest reader.
Operational truth. Incremental systems do not fail loudly. The pipeline stays green while the numbers drift. Schedule the rebuild you think you do not need.