Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Watermark-based incremental batch

When is incrementality just a watermark comparison, and where does it fail?

If data arrives append-only and is partitioned by a key that correlates with arrival time (the date folder), then "what changed?" collapses to "which are new?" Detection becomes a directory listing plus a comparison against a stored high-watermark.

done = load_watermark()                      # e.g. "2026-03-14"
for day in list_partitions("events/"):
    if day > done:
        process_partition(day)               # writes its output idempotently (overwrite)
        save_watermark(day)

There is no capture infrastructure and no state, and F is close to zero. This is why the overwhelming majority of working incremental pipelines are this and nothing more. And it is why reaching for CDC before checking whether this suffices is the characteristic over-engineering of the area.

Where it fails

It fails precisely where its assumption fails:

  • Late-arriving data lands in a partition that is already marked processed. (Time semantics studies lateness as a completeness question: when can we say a time window is complete? Here the same fact is an idempotence problem: reprocessing must give the same answer.)
  • Restatements of old partitions (a correction to last month) are invisible, because only new partitions are examined.

The standard repair: a re-processing window

Always reprocess the last N periods, overwriting idempotently. This is bounded repair at bounded extra cost. Example: with N = 3 days, each daily run reprocesses today and the 2 previous days. A GPS batch that arrives up to 2 days late is picked up. Extra cost is 3× instead of 1× on a small daily partition.

It fails silently for any correction older than N. Tuesday's upload of data for Sunday is fine; an upload for the previous month is not.

In one line: and costs almost nothing.