Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Structure becomes a variable

What makes a schema change a multi-party coordination problem with no atomic update?

Most of the mechanisms in a database or data platform treat structure as given. Storage layout, encoding, indexing, query planning and data modeling all assume that the shape of a record is a fixed input. This mechanism removes that assumption. It asks what happens when the shape itself changes, on its own schedule, independently of the code that reads it.

Four things are new here.

1. Inside a single database engine, changing a structure is a transaction: a group of steps that either all happen or none happen, and at the end everyone sees the new thing. Here you have:

  • a writer deployed at time T,
  • data written at time T that is still on disk at T + 3 years,
  • a reader compiled at T − 1 that nobody has touched since,
  • and a second reader that nobody told you about.

No transaction spans all of these. Schema change is therefore a multi-version, multi-party coordination problem, in a small and sad form. It also inherits none of the tools that make coordination of servers tractable: there is no vote among nodes (a quorum), no elected leader, no membership list. You cannot even take a census of who reads your table.

2. Performance stops mattering. Nothing here moves latency, throughput or byte counts. Looking up a by an ID instead of by position is a lookup. The costs here are measured in where the failure shows up and how much coordination is needed. They are paid by people, often people who did not make the decision. The decision is made by the , and the cost is paid by an unknown . This move of cost from the decider to someone else is the defining economics of the topic.

3. The direction of the constraint flips. In most layers the mechanism serves the data. Here the data was already written and cannot be rewritten cheaply. So the old data constrains the new schema. Evolution only ever means: "what can I change such that everything already on disk stays readable, and everything already deployed keeps working?" This is not a design freedom. It is a : the set of allowed changes must be one that, however many times you apply it, leaves old data readable.

4. The governing question is about timing and surprise. It is not "what is the schema?" It is:

"Bound" means "fixed". When the meaning of a field is fixed (at write time, at read time, at deploy time) and where a disagreement becomes visible (as a loud error, or as a quiet wrong number) are the two things to control. Two sub-questions are special cases of it:

  • If producers and consumers are tightly coupled (they must agree before any change), coordination is expensive. If they are loosely coupled (each moves alone), coordination is cheap, but surprises appear.
  • When data does not fit, should the system stop immediately and loudly, or continue with something worse than the truth (a , a default) and keep running?

A candidate rule

One candidate rule runs through this mechanism: .

Worked illustration (invented numbers). A table has 3 readers and one month of data in year one. A change touches 3 × 1 = 3 "reader-months" of exposure. In year five, the table has 12 readers and 60 months of data. The same change touches 12 × 60 = 720 reader-months, which is 240 times more. If this holds, it explains why schema evolution is easy in year one and enormous by year five. It does explain the cost of , the case for transitive checks and the buildup of dead columns. It is not clearly separate from a second rule, that an interface stays stable only when breaking it costs the producer, so it stays a candidate, not a law.