Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Backfill and cutover

Why should a backfill overlap the live pipeline on purpose rather than meet it exactly?

Backfill is the reconciliation of history with a pipeline that was designed for the present. Historical are independent, so a backfill can run wide and fast. That is also the problem: it contends with the live pipeline for the same compute, the same sink and the same source. So you throttle it, give it a separate resource pool, and schedule it off-peak.

Cutover: where correctness is decided

Cutover is the moment when the live pipeline takes over from the backfill. It is the same gap-or-overlap problem as the handoff in Bootstrap and represent.

  • The backfill covers history up to time T.
  • The live pipeline starts at time T′.
  • If T′ > T, you have a hole (a gap of missing data). This was Tariq's missing day.
  • If T′ < T, you have an overlap. An overlap is harmless only under an idempotent, keyed sink.

So the design rule is forced:

Attempting a precise seam (T′ exactly equal to T) is choosing a hard problem over an easy one.

Blue-green at table level

The mature pattern is at table level:

  1. backfill into a ;
  2. run old and new pipelines in parallel;
  3. diff the two tables until the diff is empty;
  4. swap by pointer (for example rename a view or update a catalog entry).

The swap is a metadata operation, and it is cheap. The confidence comes from the diff, not from the swap.

In one line: never aim for an exact seam; overlap deliberately and let idempotence absorb it.