What metadata forbids, and what the escapes cost
What this topic forbids
Six things cannot be done, whatever tool you buy:
- You cannot keep a second copy of a fact true for free. Coherence is the price of every catalog entry.
- You cannot update two stores atomically, so reconciliation is mandatory, not optional.
- You cannot make a negative claim from incomplete observation. This invalidates most safety uses of lineage and telemetry as usually practised.
- You cannot get durable metadata quality by asking people, because the producer and the consumer are different people at different times.
- You cannot scale metadata by bytes. It scales by object count.
- You cannot have concurrent writers without a linearization point. It exists somewhere, whether or not you designed it.
The escapes, and what each costs
| Escape | What it gives | What it costs |
|---|---|---|
| Make metadata load-bearing | Errors surface immediately, so the metadata stays correct without anyone maintaining it. The strongest move. | The metadata is now on the critical path. Its availability becomes your availability: if the catalog goes down, writes stop. |
| Make it a by-product of execution | Free and accurate metadata. | Coverage limited to instrumented engines and to code paths that actually ran. |
| Make it immutable and pointer-swapped | Atomic commits, free versions, O(1) rollback. | Space and a collector (a cleanup process). |
| Push metadata into the data as provenance columns | Derivation is verifiable instead of inferred. | Paid at data scale (per row). |
| Add freshness stamps and confidence decay to advisory metadata | Silent wrongness becomes visible uncertainty. | Almost nothing. It buys only honesty, which is why it is undervalued. |
| Standardize interchange | n × m integrations become n + m. | Lowest-common-denominator meaning. |
| For the problem of absence: a mandatory channel (deny-and-log, revoke-and-wait) | Replaces "I observed nothing" with "nothing can happen without telling me." | Possible short disruption for hidden users; operational discipline. |