Load-bearing versus advisory
Which metadata cannot be wrong without someone noticing, and how does its truth source set its decay?
Most treatments of metadata blur this distinction. It is what makes the topic manageable. Metadata splits by what breaks when it is wrong.
| Load-bearing | Advisory | |
|---|---|---|
| Role | The data cannot be read without it | The data is readable; humans are misled |
| Examples | table-to-file mapping, manifests and snapshots, partition specs, column IDs | lineage, ownership, glossary terms, tags, descriptions, usage statistics |
| Failure mode | Loud, immediate, visible as wrong results or errors | Silent, delayed, visible only as loss of trust |
| Consistency needed | Transactional with the data | Eventually consistent, often never |
| Decay rate | Cannot decay: an error shows up on the next read | Decays continuously; bounded only by how often someone checks |
| Who maintains it | The writer, mechanically, as part of the commit | Nobody, unless it is automatic |
| Cost of being wrong | Paid by the system | Paid by the next person |
The trajectory of this whole field over the last decade is converting advisory metadata into load-bearing metadata, because load-bearing metadata is the only kind that stays true.
Example with statistics. If "number of rows" is an advisory note refreshed by a nightly ANALYZE job, it goes stale between runs. If the same number is written into the table's manifest (the file list) at commit time, it cannot go stale: the commit that wrote the file also wrote the number.
A second cut: where does the claim get its truth?
- By construction. Written at the moment of the fact, by the process that caused it. Examples: manifests, snapshot logs, statistics written at commit time, lineage captured while a job runs. It cannot be stale. It can only be incomplete (some things never emit it).
- By observation. Derived afterwards from logs, queries or crawls. Examples: query-log mining, access logs, crawled schemas, popularity ranking. It is always stale by one sampling interval and limited to what was observed.
- By assertion. A human typed it. Examples: glossary terms, ownership, sensitivity labels, descriptions. It is correct at the moment of typing, then decays steadily, and gives no signal at all when it becomes false.
The decay rate ranks strictly: construction ≪ observation ≪ assertion. The mechanisms here follow almost the same order, from the top (load-bearing and constructed) to the bottom (advisory and asserted). That order is also the difficulty gradient of the topic.