Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Provenance annotations

How does row-level provenance stay cheap, and what does it make possible?

means a record of where each piece of data came from: which source, which ingestion time, which pipeline version. It is different from everything else here in one way: it is per record, so its storage cost is proportional to the number of rows, not to the number of assets. It is the only with data-scale cost, and that one fact decides its design.

How it stays cheap

A batch shares one ingestion timestamp, one source identifier and one pipeline version. Columns holding such values compress to almost nothing with (store value × count instead of repeating the value) or (store each distinct value once and use small numbers to refer to it).

The cost is real but small. It collapses to nothing if you demote provenance from row level to file or level, which is the usual trade: accuracy against space.

What it buys

  • Row-level provenance makes the unit of correction a set of rows rather than a whole partition. That is the difference between surgical reprocessing and a full backfill when pipeline version 14 turns out to be wrong.
-- Which rows did the bad version write?
SELECT count(*) FROM orders_enriched WHERE _pipeline_version = 14;
-- Reprocess only those rows (assuming a source column records the origin).
  • More basically, provenance is the only mechanism here that makes derivation verifiable rather than inferred. is a claim about what code does. Provenance is a record of what execution did. When the two disagree, provenance is right.

Provenance is the only metadata priced per row, and