Summary and key takeaways

Load-bearing versus advisory

Truth source ranks decay: by construction ≪ by observation ≪ by assertion.

Key mechanisms, one line each:

• Asset identity. A stable meaningless ID with late-bound names. A path as identity makes every rename a silent delete.
• Metastore. An OLTP database sized by number of objects, not bytes. Small files hurt it before they hurt scans.
• A partition is a proof about files you have not read. In folder-encoded partitioning the proof cannot be verified.
• Manifest trees put metadata in the object store as immutable files. Pruning becomes hierarchical, and the small-file problem recurs one level up.
• The commit point. One compare-and-swap on a catalog pointer makes many file changes atomic. The catalog is the table's linearization point and sets one commit per round trip as the throughput ceiling.
• Reconciliation. Write data before metadata; delete metadata before data. Orphans are harmless; dangling references are not. Age thresholds assume something unobservable.
• Statistics. Scan-based are stale where change is fastest; write-time are exact and mergeable, which is why sketches live in manifests.
• Provenance. The only per-row metadata, and the only derivation evidence.
• Versioning. Branch, read and rollback are O(1); merge is not. Retention is an economic limit.
• Schema IDs are immutable, so caches need no invalidation.
• Collection. Pull, static parsing and runtime push fail differently. Coverage, not accuracy, is the quality metric.
• Lineage is sound but not complete. It can prove a dependency, never its absence. Use a mandatory channel for absence claims.
• Labels propagate along lineage, but inherit its incompleteness, so unlabelled must fail closed.
• Ownership decays fastest; derive it from load-bearing facts and resolve it to a group.
• Usage ranking is a feedback loop; add a damping term. Telemetry cannot prove disuse.
• Audit is not telemetry. Audit is complete, tamper-evident and long-lived, and cannot answer per-row questions.
• Catalog cache TTL is an isolation setting. Freshness stamps turn silent wrongness into visible uncertainty.

Four rules:

1. Metadata that is not a by-product of execution decays to noise.
2. A claim held outside its system is a cache with no invalidation channel: make it load-bearing, give it a TTL, or accept it is wrong.
3. An incomplete observer can prove existence, never absence.
4. Order writes across two stores so the survivable inconsistency is the one that happens.

Numbers to remember:

• Metadata is 10⁻³ to 10⁻⁶ of data size, but grows with object count.
• Hourly partitions for three years = 26,280 per table.
• One successful commit per catalog round trip: 100 ms means at most 10 commits per second.
• Column-level lineage graphs are 100 to 1,000 times larger than table-level ones.
• Annual owner turnover of 20–30% leaves an ownership field correct after three years only 34–51% of the time.
• Join-estimate errors compound multiplicatively: 3× per join over 4 joins is 81×.