Identity by name and by ID

Why is rename a drop plus add under names, metadata-only under IDs, and why must an ID never be reused?

Identity by name

Identity is a string.

• Reorder and insert become free: a reader finds the field by its name, wherever it is.
• Rename becomes drop plus add. The values in old data are stored under the old name. Under the new name they cannot be found.
• Matching rules become a correctness problem. Upper-case versus lower-case, extra spaces and Unicode normalization (the same character written with different byte sequences) all decide whether two strings are "the same name". This sounds pedantic until two engines disagree about Amount and amount: one finds the column, the other finds nothing.

Ravi's weight to weight_kg change is exactly this: under name identity, the old data's weight values became unreachable under weight_kg, and the readers who asked for weight found no weight in new records.

---

Identity by ID

Identity is an integer assigned once, when the column is created. The name and the position both become ordinary mutable attributes that hang off the ID. The ID never changes.

• Rename is metadata only. Change the name attribute, keep the ID. All files already written still say "column 7", and column 7 is now called weight_kg.
• Drop is safe: remove the ID from the schema; old files that contain data for it are simply ignored.
• Reorder is free.

This is what table formats converged on. The reason is not ergonomics. On immutable files you cannot rewrite the header of old files. So every old file must stay interpretable under the current schema, forever. Only a stable identifier gives you that.

The invariant: never reuse an ID

ID identity works only under one rule: an ID is never reused. Suppose column 7 was weight and you drop it. Later someone adds a new column delivery_fee and the system recycles ID 7. Now every file written before the drop is read as if column 7 were delivery_fee. The bytes are the same, the meaning is new, and no error appears anywhere.

This is the general rule of a monotonic identifier that must never be recycled. It appears in other places in data systems as well: fencing tokens and epochs (numbers that grow with each new leader, so a stale leader can be refused), and version chains (the numbered versions of a record in a multi-version database).

Identity determines the operation set. Give every column an immutable ID and never reuse one, or accept that rename is a rewrite.

---

Comparing the three rules