Interview questions
What counts as the same field
1A field was renamed, and a downstream dashboard dropped to zero with no error. Why?
When fields are identified by name, a rename is a drop plus an add. The reader asked for the old name, found nothing, and produced null or zero.
Read moreIdentity by name
2Why did open table formats choose column IDs over column names?
Their files are immutable, so old files cannot be rewritten when the schema changes. Only a stable ID lets every old file stay readable under the current schema, and makes a rename a pure metadata change.
Read moreIdentity by ID
3What goes wrong if a column ID is reused after its column was dropped?
Old files are read as the new column: the same bytes with a new meaning, and no error. IDs must never be recycled.
Read moreNever reuse an ID
4Why is a CSV file without a header so dangerous when the schema evolves?
Fields are identified by position, so inserting or dropping a column shifts every column after it. Neighbouring numeric columns still parse, so you get wrong values rather than errors.
Read moreIdentity by position
Where structure is bound
5Is schema-on-read really schema-free?
No. It is N private schemas, inferred separately by N readers, none documented. The work is not removed; it is multiplied and hidden.
Read moreThe common description is wrong
6Why can an inferred schema not be trusted for production column types?
The inferred type depends on the sample (how deep it looks, the order of files, how much it reads), so the same data can yield different schemas. Freeze the inferred schema, and treat any later difference as an alert.
Read moreType instabilityThe useful inversion
7How do you allow open-ended customer-specific fields without a deployment for each new one?
A variant (semi-structured) column, with the commonly used paths shredded into typed sub-columns, plus a promotion rule: a field present in more than X% of records for more than Y days becomes a declared column.
Compatibility
8What is the difference between backward and forward compatibility, and which lets you deploy writers first?
Backward compatible: a new reader can read old data, so you deploy readers first. Forward compatible: an old reader can read new data, so you can deploy writers first.
9Why do you need a transitive compatibility mode?
Readers can be more than one version behind. Changes from A to B and from B to C can each be safe while A to C is broken.
Read moreTransitive modes
10Which single schema change is safe in both directions?
Adding an optional field with a default. Almost every other change breaks in at least one direction.
11How do you safely rename a column that consumers you do not know about may be using?
Expand and contract: add the new column, write both, backfill the new one, migrate readers, verify that nobody reads the old one (which needs telemetry on reads), and only then drop it.
Read moreWhy rename needs several steps
12Why do mature tables accumulate dead columns?
The final drop requires proof that nobody reads the column. Without read telemetry or lineage, that proof is impossible, so the contract step never happens.
Read moreThe hard step is the last one
13What is the difference between type promotion and type coercion?
Promotion is a widening that loses nothing and maps every value uniquely (int32 to int64), so it is safe to automate. Coercion is a conversion at read time that may lose information, and whose failures become a null, an error or truncation.
Read moreType promotion versus coercion
14Why is 'why are 30% of these values null?' often unanswerable?
A field can be absent, present and null, or present and filled with a default: three different facts. Most systems store all three as the same null.
Read moreThe three-state problem
15What is dangerous about a default applied by the reader?
Changing it retroactively changes the meaning of all historical data, with no data changed, no version bumped and no compatibility check run.
Read moreWhere does the default live?
16Why is adding a value to an enum a breaking change?
The types are unchanged, but consumers that handle every case explicitly (CASE expressions, joins, constraints, hard-coded dashboards) mishandle the new value. The set of allowed values is part of the interface.
Read moreEnum and symbol-set evolution
Enforcement
17Why does a schema registry exist, and what does it cost?
It reduces N × M producer–consumer agreements to N + M. The cost is a metadata service in the data path, so consumers should cache schemas for decoding, and only registering a new version should need the registry synchronously.
Read moreWhy a registry existsAvailability
18Why is a schema diff in CI the highest-leverage control?
It catches a breaking change while it is only a branch and has produced no data. Catching it at deploy costs about 10 times more, and at run time about 10 times more again.
Read moreCI-time schema diff
19When is permissive parsing, accepting bad records by guessing, acceptable?
Only when consumers can detect the degradation themselves, and with a counter of violations that someone is alerted on. Otherwise, reject.
Read moreThe decision variable
20Why is a 40% dead-letter rate not just 'bad data'?
It is an undetected schema change. Sending it to a dead-letter queue turns it into a slow bleed at full volume. Escalate to stopping the pipeline when the rate crosses a threshold.
Evolving at scale
21How can a table's partitioning change without rewriting the table?
Record the partition spec per file in metadata, as a declared function of a column, such as day(ts). The planner applies whichever spec was in force when each file was written.
22Why can SELECT * be dangerous inside a pipeline?
An additive change upstream becomes a schema change at every hop downstream, with no code change and no review.
Read moreSELECT * at every hop