Schema Management & Evolution
What happens when the shape of your data changes on its own schedule, and nobody can change the readers, the writers and the old data at the same moment?
Who this mechanism is for: a developer who writes services, reads and writes JSON, CSV or database tables, and has probably renamed a column or a field at least once. You do not need to know any data-platform internals. You do not need to know what a schema registry, a table format or a data contract is. Every technical term is explained the first time it appears, and again in the glossary at the end.
The story we will follow
Parcelo is a small company that tracks parcels for online shops. Every time something happens to a parcel (picked up, scanned at a depot, out for delivery, delivered), a service called tracking publishes a shipment event. Several other programs read these events: a warehouse app, a mobile app, a nightly job that loads events into a big table, a finance dashboard, and a CSV export for a partner company.
Ravi is the backend engineer who owns the tracking service. Sara is the analytics engineer who owns the finance reports. They sit in different teams. Ravi's job is to ship features. Sara's job is to make the numbers right. During this mechanism Ravi will change the shape of his events several times, in small and reasonable ways. Each change will break something he cannot see.
Each article below opens with a problem that looks like a bug in someone's code. Each one is really a consequence of how structure, time and coordination work. At the end of each article we go back and solve the problem with the mechanisms of that part.
This is the main idea of this whole mechanism:
How this mechanism is organized
- 1
A new kind of problem
Why is changing a schema different from changing code, and what four questions organize the whole topic?
CaseA rename, and revenue went quiet for nine days
- 2
Identity
How does a reader decide which bytes belong to which field, and why does that choice decide which changes are possible?
CaseA column inserted in the middle of a CSV
- 3
Where structure is bound
Is it better to check the shape of data when writing it or when reading it? What do inference and "variant" columns do?
CaseTwo dashboards, two numbers for the same week
- 4
Compatibility
Which changes are safe, in which order can you deploy, and where do types, defaults, nulls and enums hide traps?
CaseThree harmless changes, three silent breaks
- 5
Enforcement
Who checks the rules (registry, contracts, CI), and what should a pipeline do when a record breaks them?
CaseNo events for 40 minutes, then 40% missing
- 6
Evolution at scale
How do you live with many schema versions at once, change partitioning, and keep a change from spreading through a chain of derived tables?
CaseThree years of files, and nobody dares touch the table
- 7
What schema evolution forbids
What can never be done, and what does each escape cost?
After that: a summary, hands-on labs, interview questions and a glossary.
One request: read the one-line summaries in italics at the end of each section. They contain the rule you should remember.
What do I do, and what does each option cost?
Can I see it myself?
Positional versus named CSV
Read a headerless CSV by position, insert a column in the middle, read it again; then add a header and read by name.
Break itPythonRename is drop plus add, in real Parquet files
Two Parquet files, one with weight and a newer one with weight_kg, read together with union_by_name; then COALESCE.
MeasurePython with DuckDBInference is a function of the sample
Infer a schema from the first 1,000, 18,000 and 20,000 records of a file whose postal_code turns from integer to string, in file order and shuffled.
MeasurePythonBuild a tiny schema-diff gate
A compatibility classifier for schema changes and ten test cases, checked against the last version only and against every earlier version.
MeasurePythonReader-side default rewrites history
1,000 stored events without delivery_priority, counted under a reader default of STANDARD and then EXPRESS; then with the default written at write time.
MeasurePythonDead-letter rate threshold
100,000 records; a schema change makes 40% invalid halfway. A pure dead-letter queue against a 2% windowed halt threshold.
SimulatePython
Can I reason about it under questioning?
Interview questions
Short questions on schema evolution, such as field identity, schema-on-read, compatibility modes, renames and enums, registries and contracts, and partition evolution, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on schema evolution, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Masking moves failure
Masking a failure moves it from a channel with a schema to a channel without one.
Guarantees have scope
Atomicity is bounded by the shared commit substrate.
Coordination cannot be made cheaper
Coordination is the only thing you cannot make cheaper.
Which way the error points
Designate one write
Atomicity over n items is atomicity over one designated item.