Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Metadata interchange and caching

What does a metadata standard really solve, and why is a catalog cache TTL an isolation decision?

Metadata interchange

The standard argument is correct: with n producers and m consumers, you need integrations without a shared format and n + m with one. Example: 8 tools that emit and 5 tools that read it: 8 × 5 = 40 integrations versus 8 + 5 = 13.

But the harder truth is that the wire format is the easy part.

  • Identity conventions are hard: the same asset must be named identically by every emitter. This returns to the first section of The addressing layer.
  • Event semantics are harder still: what counts as a run, a dataset, a column mapping, a failure?

Standards converge on transport and diverge on meaning. So what you actually get is a lowest-common-denominator graph, where the richest producers are flattened to what the poorest can express. That is current engineering reality and it is improving slowly. It is not a solved problem, and it should not be filed as a permanent constraint.

Also: interchange is a push protocol from producers, so it inherits the coverage ceiling. A standard does nothing about the tool that emits nothing, which is exactly the tool that creates your gap.

Metadata caching, and metadata about metadata

Two short items complete the picture.

Client-side catalog caches

Clients answers, because planning latency demands it. The cache's then becomes a visible consistency property: a reader holding a stale pointer reads an older .

Example: with a 60-second TTL, a reader may see the table as it was up to 60 seconds ago, even after a writer committed. A dashboard that joins two tables, each cached at a different moment, can see them at different versions.

Freshness stamps and confidence decay

For , where no channel exists, a partial escape is to record when the claim was last confirmed and to fade the displayed confidence with age. For example: "Owner: Sam (last confirmed 14 months ago)", shown in a warning colour.

This does not make the description true. It converts silent wrongness into visible uncertainty, which is the difference between a user being misled and a user who knows to check. It costs almost nothing and is almost never done.