What schema evolution forbids
What this mechanism forbids
These are not engineering gaps. They are limits.
- Atomic structural change across independently deployed parties. No protocol provides it. This is permanent.
- Narrowing anything (type, domain, nullability) against written data without a data check or a rewrite.
- Knowing your consumer set without making consumers register. It is unverifiable by construction.
- Detecting semantic change with any structural mechanism. Cents to dollars passes every gate ever built.
- Renaming under name-based identity without expand/contract, in any engine.
- Reversing a lossy coercion, ever. The original is gone at the moment the coercion was applied.
- Completing the contract phase of a rename or drop without read observability.
Escapes and their prices
| Escape | Buys | Costs |
|---|---|---|
| ID-based column mapping | free rename, reorder, drop | ID assignment plus a never-reuse discipline in both the file format and the catalog |
| Add-optional-with-default only | order-independent deploys | schemas only grow; dead columns accumulate with no collector |
| Variant column | open payloads without deploys | late binding, weaker compression and pruning, needs a promotion policy |
| Envelope version plus reader branch | arbitrary incompatible change | permanent dual-path code, forever |
| Full transitive mode | safety against arbitrarily stale readers | removes most operations; drop becomes near-impossible |
| Rewrite the data | one clean schema | full table rewrite, cache invalidation, a large data-rewrite job |
| Dead-letter | output correctness | completeness, plus a queue that requires a watcher |
| Permissive plus counter | completeness | correctness; safe only if the counter is alerted on |
| Contracts | ownership and a consumer list | organizational cost; effective only if breakage is priced to the producer |
| CI schema diff | cheapest possible detection | the schema must be derivable from code |