Summary and key takeaways
The governing question. At what moment is a field's meaning bound to a reader's expectation, and where does the mismatch surface? Two sub-dials: coupling versus coordination cost, and fail fast versus fail degraded.
The four questions and their answers:
| Question | Answer in one line |
|---|---|
| Where is structure bound? | Validation is conserved. Declare a required core once; keep a scoped late-bound payload; freeze inferred schemas and use later inference as an alarm. |
| What is "the same field"? | Use immutable IDs and never reuse them. Position is silent corruption; name makes rename a drop plus add. |
| Which changes are legal? | Add optional with default; widen; never narrow, never add required. Rename is expand/contract. The mode you need depends on a deployment order you do not control, so full transitive is the honest default. |
| Who enforces, and what on violation? | Registry (N+M agreements, cached off the data path), contracts (owner, consumers, notice), CI diff (earliest detection), dead-letter with a watcher and a rate threshold. |
Key rules to remember:
- Compatibility belongs to a (writer, reader) pair, and the registry only checks a proxy.
- Forward compatibility depends on reader parsers ignoring unknown fields.
- Promotion is a provable widening. Coercion is a lossy parser. Only promotion belongs in an automatic path.
- Absent, null and defaulted are three facts. A reader-side default rewrites history silently.
- Tightening a constraint is verify-then-declare.
- Adding an enum value is structurally safe and semantically breaking.
- You migrate the schema, not the data; every past schema must stay readable from the current one.
- Partitioning is evolvable only if it is a declared function of a column, recorded per file.
SELECT *turns a safe addition into a change at every hop. Lineage plus usage telemetry answers "who breaks?".- Reject when the consumer cannot notice, coerce only with a counter, and escalate on rate.
Detection costs: merge (failed build), deploy (blocked release), runtime (incident plus bad data), roughly ten times more at each step.
What no mechanism sees: a change of meaning with an unchanged type (cents to dollars).
Two rules about who decides and who pays: the late-bound decision lands on the party with the least information about intent; an interface stays stable only when breaking it is expensive for the party who breaks it.
The tests of any schema claim: Who is the consumer set, and can you list it? Where is each default stored? What happens to a record that does not fit: is the failure loud or silent, and who pays?