Lineage granularity
What do job-, table- and column-level lineage each cost and tell you?
There are three levels. Their cost and value scale very differently.
| Level | Where it comes from | What it tells you | Size of the graph |
|---|---|---|---|
| Job-level | the scheduler, free | what ran before what (scheduling dependencies) | O(jobs) |
| Table-level | the leaves of the execution plan: which tables were read, which written | "what breaks if I change this table" | O(tables); manageable |
| Column-level | tracking derivation through the operators: expressions, aggregations, CASE statements, UDFs | "does this change affect that number" | two to three orders of magnitude larger |
- Job-level is cheapest and least informative. It says what ran before what, not what depends on what.
- Table-level is the common case. Any engine that can report its inputs and outputs can produce it.
- Column-level needs either engine internals or a full-fidelity SQL parser for each dialect. That is the reliable dividing line between having lineage and having lineage that answers questions. What it buys is the question people actually ask, plus sensitivity-label propagation (below), which is the only genuinely load-bearing use of lineage.
Lineage is a graph, so graph problems follow
Because lineage is a graph, it inherits the graph's mechanics: transitive-closure queries, cycle handling (cycles do occur, through feedback tables and reconciliation loops) and fan-out explosion. "Everything downstream of the raw events table" is the whole warehouse. So impact analysis without ranking returns, for example, 3,000 assets, which is the same as no answer. Ranking needs usage data (The social layer), which is why the relationship layer and the social layer cannot really be separated.