Metadata, Catalog & Lineage
What can a system prove about its data without reading the data, and how does it know that proof is still true?
Who this mechanism is for: a developer who writes code, runs SQL queries and has used files in cloud storage, but has never built or operated a data catalog, a table format or a lineage tool. You do not need database internals. Every technical term is explained the first time it appears, and again in the glossary at the end.
The story we will follow
Hana is a data engineer at Trailmix, an online shop that sells hiking gear. Trailmix keeps its data in cloud object storage as Parquet files (a compressed column-based file format), runs SQL on it, and has about 2,000 tables. Diego leads the analytics team. Priya is the compliance officer who answers to regulators about customer data.
Nothing in Trailmix is broken in the usual sense. No file is corrupt and no server is down. But during one quarter, Hana will meet five problems, one at the start of each article below. Each problem looks like a bug. Each one is actually a consequence of how information about data behaves. At the end of each article we go back and solve it.
The main idea of this whole mechanism:
How this mechanism is organized
- 1
Claims about data
What is metadata, why is it different from data, and which metadata can be wrong without anyone noticing?
CaseA catalog page that is wrong in every field
- 2
The addressing layer
How does a system know which files make up a table right now, and how do two writers commit safely?
CaseA rename, and the table's history vanished
- 3
The description layer
How are statistics, per-row origin, versions and schemas stored so that they stay true?
CaseA terrible plan on the fastest-growing table
- 4
The relationship layer
How is lineage (what feeds what) collected, how complete is it, and what can it safely be used for?
CaseNo path exists, and the emails reached the partner
- 5
The social layer
Who owns a table, how do people find the right table, and how do usage logs and audit logs differ?
CaseForty revenue tables, and the top one deprecated
- 6
What metadata forbids
Which rules explain every mechanism above, and what does each escape cost?
CaseThree governance platforms: which promises hold?
After that: a summary, hands-on labs, interview questions and a glossary.
One request: watch for the two questions that return in every section. Is this metadata load-bearing (the system breaks if it is wrong) or advisory (only people are misled)? And: where does the claim get its truth: from the system itself, from observation, or from someone typing it?
What do I do, and what does each option cost?
Can I see it myself?
Planning time versus partition count
Read the metadata of 1,000 to 1,000,000 partitions from SQLite, with and without a filter, and size the planning slots with Little's Law.
MeasurePythonA file in the wrong folder gives wrong answers, silently
Copy one day's Parquet file into the next day's Hive folder and read the table; then store the date inside the file and count mismatches.
Break itPython with DuckDBThe pointer swap: two writers, one winner
Two writers commit table versions through a pointer: with an exclusive create (compare-and-swap), then with a plain write.
Break itPythonRead the statistics inside a Parquet file
Read Parquet footer statistics with parquet_metadata: files skippable for a one-hour range, sorted against random; merge two files' statistics by hand.
MeasurePython with DuckDBLineage coverage: static parse against runtime log
Lineage for 20 small jobs by parsing their SQL and by recording what DuckDB actually scanned: edges found by each.
MeasurePython with DuckDB, sqlglotThe ranking feedback loop
52 weeks of catalog search where most users take the top result: does the better table ever win, and what certification bonus does it take?
SimulatePython
Can I reason about it under questioning?
Interview questions
Short questions on metadata and lineage, such as load-bearing metadata, catalogs and commits, statistics and versioning, lineage and labels, ownership, usage and audit, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on metadata and lineage, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Early binding vs. late binding
When structure is committed; determines where the failure surfaces, at write time or read time.
Consistency vs. availability vs. latency
For example: quorums, replication modes.
Mutability vs. immutability
In-place update vs. append-only + GC; buys reproducibility and cheap concurrency, pays in space and janitorial work.
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Absence is not evidence
Absence of an event is not absence of a failure.
Granularity mismatch
Granularity mismatch creates amplification.
Fixed cost per operation
Fixed cost per operation implies a natural batch size.
Immutable storage grows indirection
An immutable substrate grows an indirection layer plus a collector.
Sound approximations
When exactness is too expensive, use a sound approximation.
Show all 11 principlesShow fewer
Variance costs more than volume
Variance costs more than volume.
Which way the error points
Designate one write
Atomicity over n items is atomicity over one designated item.
Masking moves failure
Masking a failure moves it from a channel with a schema to a channel without one.
Guarantees have scope
Atomicity is bounded by the shared commit substrate.
The end-to-end argument
Guarantees compose to the weakest link, and only endpoint guarantees are real.