Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Collection: push, pull and coverage

How does metadata get collected, and why is coverage the quality number that matters?

How arrives matters more for its quality than any other design choice. There are three ways.

  1. Crawling and scraping (pull). An outside process walks through the systems at intervals. Cost: O(assets) per cycle. Freshness equals the interval. Coverage equals whatever is reachable from outside. It breaks on anything that cannot be seen from outside.
  2. Static parsing of code. It reads SQL and code without running anything, so it covers paths that never ran. But it fails on dynamic SQL (SQL text built while the program runs), user-defined functions (UDFs), external configuration, and anything assembled at runtime. Its bias goes both ways: it over-reports edges that exist in code but never run, and under-reports edges created at runtime.
  3. Runtime capture (push). The engines themselves report what they actually read and wrote. It is exact for what ran, and it misses everything that did not run. Its cost is instrumentation of each engine, which is the real obstacle.
// A runtime lineage event, emitted by the engine when a job finishes
{
  "job":    "export_partner_daily",
  "run_id": "2025-03-07T02:00:00Z",
  "inputs":  ["asset:7f3a-91c2"],
  "outputs": ["asset:c41e-0b77"]
}

and runtime capture fail in opposite directions. That is why mature systems run both and reconcile them rather than choosing one.

The quality number that matters: coverage

Neither accuracy nor granularity is the key metric. It is coverage: the fraction of your computation that emits metadata at all.

One team's unconnected notebook, or one vendor ETL product, makes a permanent structural hole. And the hole does not announce itself: the graph just looks complete and is not.

This is the choice between push and pull deciding an information-quality outcome rather than a performance outcome.

Coverage, not accuracy, is the quality metric for collected metadata, and it is capped by your least-instrumented engine.