Claims, not bytes
Why do facts about data fail quietly, and what governs metadata instead of bytes?
Everything in the earlier parts of a data system manipulates bytes that are the data. This mechanism manipulates claims about data. The difference is a difference in kind: a claim can be wrong while the system keeps running.
- A corrupt data page fails loudly at read time.
- A wrong owner field, a stale statistic or a lineage edge that no longer exists fails silently, at human speed, months later, inside someone else's decision.
What stops mattering: bytes, file layout, latency budgets in microseconds. Metadata is tiny compared with data (10⁻³ to 10⁻⁶ of the footprint). Nothing here is limited by bandwidth.
What starts mattering instead: cardinality and staleness.
- Cardinality. The volume of metadata follows the number of objects (files, partitions, columns, jobs, runs), not the number of bytes. Example: a petabyte stored in 200 files is a trivial metadata problem. A terabyte stored in 20 million files is a catastrophic one. When metadata grows large enough, it becomes a database workload of its own: it has hot spots, write contention and a consistency model. That recursion is the defining feature of this mechanism: the catalog is a database, and it is usually the least-engineered database in the stack.
- Staleness. Metadata is a second copy of a fact whose first copy lives somewhere else: in the storage layer, in the query engine, or in someone's head. Every second copy is a cache. (The cost did not disappear. It moved into a coherence problem: keeping the copy equal to the original.) And unlike an ordinary cache, most metadata has no invalidation channel. The change happens in a system that does not know the catalog exists. A team renames a column; nothing tells the glossary. Someone deletes files through the storage API; the metastore still claims the files are there.
A third constraint is organizational, and it is the one that actually kills metadata systems. The person who produces metadata is almost never the person who uses it. Writing a description costs effort now. The benefit goes to a stranger later. This imbalance has one durable solution, and the whole modern data stack arrived at it: stop asking people to write metadata. Make it an unavoidable by-product of running the work.
The governing question of this mechanism:
What can the system prove about the data without reading the data, and how does it know that proof is still true?
And the one-sentence summary: