Scenarios

You cannot list every consumer or deploy them all at once, and with fields identified by name a rename is a drop plus an add.

Registry subject consumers, mining of query logs, and a contract with a registered list of consumers. Unknown consumers set the minimum notice period.

Expand and contract: add weight_kg (optional, with a default), write both fields for at least one full retention window, backfill, notify teams and track each one's migration, verify zero reads of weight for 30 days, then drop it.

First make sure every consumer has an explicit branch for unknown values, then add the value. Or move the statuses into a reference table.

Add a schema-diff gate in CI and transitive compatibility checking. The process takes months, the schema temporarily has duplicate fields, and without read telemetry the final drop never happens: treat telemetry as a prerequisite, not a bonus.

A fixed, required core (tenant, event type, timestamp, key, schema version), validated on write, plus a variant column for each tenant's own fields, with frequently used paths shredded into typed sub-columns.

A promotion rule: for example, a field in over 60% of one tenant's records for 14 days becomes a declared column. Freeze each tenant's inferred payload schema and alert on drift, rather than widening it automatically.

Variant columns compress and prune less well, and every variant path a dashboard reads is a promise bound late, unless promoted. Dashboards should read only the core and promoted columns.

Without a promotion rule, within a year most data lives in the variant, and you have rebuilt schema-on-read.

1. A producer change made records invalid, and a dead-letter queue is silently absorbing them.
2. Permissive parsing is coercing values to null.
3. A default applied by readers changed.

Dead-letter queue depth and growth, violation counters, the null rate per column over time, and row counts against the source.

An unwatched dead-letter queue with no rate threshold.

A violation counter; a threshold over a window (for example, more than 2% of the last 1,000 records halts the pipeline); freshness and volume expectations in the contract; and an owner who gets paged. 40% bad is a schema change, and it needs a stop, not a queue.

Fail closed (ingestion stops, data stays valid) or fail open with a cache (ingestion continues, schemas unvalidated).

Keep the registry off the data path: producers cache schema IDs and schemas, consumers cache what they need to decode, and only registering a new version needs the registry. Registration then happens in CI and deployment, a few times a month, so an outage blocks releases but not ingestion. For a genuinely new schema during an outage, fail closed for that producer only.

A registry that is 99.9% available is down about 8.8 hours a year; if every message needed it, ingestion would inherit all of that.

The partition value must be a declared function of a column, such as day(ts), recorded in metadata per file, not encoded in file paths. If so, add hour(ts) for new files; old files stay daily.

Queries spanning the boundary prune well on new files, but read 24 times more per hour of range on old ones.

If most queries hit the last 90 days, rewrite only that range, as a bounded job. If queries spread across all three years, a full 5 TB rewrite may be justified, but it changes file identity and invalidates caches.

The change is a rewrite plus a contract change with every reader of those paths. Either way, performance differing by era is permanent until old files are rewritten.

Machine-checked structural compatibility, turning N × M agreements into N + M. Enough when producer and consumers are one team and changes are additive.

They add what a registry cannot: semantic types and units, an owner, a registered list of consumers, freshness and volume expectations, and a notice period. Needed when teams differ, because the main risks are changes of meaning (cents to dollars) and consumers nobody knows about.

Contracts work only if breaking them costs the producer; without that incentive, they get bypassed.

A registry plus a CI diff plus a contract with a consumer list, for any table read by other teams; and tie producers' on-call or service-level penalties to breaking their consumers.

Requires proof the columns are unused: column-level lineage combined with usage telemetry, for example no reads in 90 days. With columns identified by ID, dropping is a metadata change and old files stay readable.

It is not a safe widening: the scale rescales values and precision can narrow. Do it as expand and contract, with a new column and a backfill, not in place.

A schema-only migration keeps every old version alive, so transitive compatibility must continue to hold.

Do not drop: the contract step cannot be decided, and the first investment is observability of reads.

Coercion turns a type error into a value, and values have no error channel, so the failure moves into the data, where it looks normal: a coerced null is indistinguishable from a genuinely missing value. And it is irreversible, since the original is gone once converted.

Allow coercion only for provably safe promotions (int32 to int64); reject everything else to a dead-letter queue that someone watches, with a violation counter and a rate threshold.

Can each consumer detect degraded data on its own? If not, rejection is mandatory.