Skip to content
About
Main content

What metadata forbids, and what the escapes cost

What this topic forbids

Six things cannot be done, whatever tool you buy:

  1. You cannot keep a second copy of a fact true for free. Coherence is the price of every entry.
  2. You cannot update two stores atomically, so is mandatory, not optional.
  3. You cannot make a negative claim from incomplete observation. This invalidates most safety uses of lineage and telemetry as usually practised.
  4. You cannot get durable metadata quality by asking people, because the producer and the consumer are different people at different times.
  5. You cannot scale metadata by bytes. It scales by object count.
  6. You cannot have concurrent writers without a linearization point. It exists somewhere, whether or not you designed it.

The escapes, and what each costs

EscapeWhat it givesWhat it costs
Make load-bearingErrors surface immediately, so the metadata stays correct without anyone maintaining it. The strongest move.The metadata is now on the critical path. Its availability becomes your availability: if the catalog goes down, writes stop.
Make it a by-product of executionFree and accurate metadata.Coverage limited to instrumented engines and to code paths that actually ran.
Make it and pointer-swapped commits, free versions, O(1) rollback.Space and a collector (a cleanup process).
Push metadata into the data as columnsDerivation is verifiable instead of inferred.Paid at data scale (per row).
Add freshness stamps and confidence decay to Silent wrongness becomes visible uncertainty.Almost nothing. It buys only honesty, which is why it is undervalued.
Standardize interchangen × m integrations become n + m.Lowest-common-denominator meaning.
For the problem of absence: a (deny-and-log, revoke-and-wait)Replaces "I observed nothing" with "nothing can happen without telling me."Possible short disruption for hidden users; operational discipline.