The arithmetic that governs everything

Below what change fraction does incremental processing beat a full rebuild?

Before any capture mechanism, we ask whether incremental processing applies at all. It is a cost comparison.

• Full recompute costs about S · c. S is the state size (rows). c is the per-row cost on a sequential scan, the cheapest access pattern that exists.
• Incremental costs about Δ · S · c′ + F. Δ is the changed fraction (for example 0.02 = 2% of rows). c′ is the per-row cost of a random-access merge into existing state. F is a fixed overhead: offset tracking, state storage, metadata commits, capture infrastructure, and the human work of keeping all of it honest.

Two things follow immediately.

1. The merge penalty c′/c. You gave up scanning and bought seeking, rewriting and small-file churn. On an immutable columnar substrate (files that cannot be edited), c′/c is commonly 10–100×, because touching one row rewrites its whole file. This is the same granularity mismatch we meet everywhere in storage: the unit you change (a row) is much smaller than the unit the medium can change (a file of 128 MB). So incremental processing stops winning long before Δ reaches 100%. The crossover is often somewhere in the 1–10% range, depending on how the underlying storage handles small mutations.

2. The fixed cost F does not shrink with Δ. At small scale incremental is strictly worse: you pay the complexity and receive nothing. Most incremental pipelines that should never have been built are explained by someone ignoring F.

The break-even formula

Set the two costs equal: S·c = Δ·S·c′ + F. Solve for Δ:

Read it in two steps. The factor c/c′ is the reciprocal of the merge penalty. The factor (1 − F/(S·c)) is a discount for the fixed cost.

So with a 20× merge penalty the crossover sits near 5%. A table where a fifth of the rows change daily should probably just be rebuilt. This is exactly Tariq's drivers table.

The claim "incremental is more efficient" is folklore in this form. The defensible version is:

Incremental converts a cost proportional to state into a cost proportional to change plus a constant. That trade is only good when state is large, change is small, and the constant is spread over many runs.