Rename is drop plus add, in real Parquet files

Measure your machineNeeds Python with DuckDB

Problem

When fields are identified by name, a rename is not a change to a field: it is the removal of one field and the addition of another. Readers that merge old and new data by name see two columns, each empty for the other half of the data, and anything that aggregates the new name silently ignores the old rows. Expand-and-contract makes the transition explicit: write both, migrate readers, then remove the old one.

Use DuckDB. Write two Parquet files, one with a column weight and one (newer) with a column weight_kg. Read both with union_by_name. Then compute COALESCE(weight_kg, weight) to see the read-side workaround that expand/contract formalizes.

The question. What does a by-name read of the two files return, and what happens to a sum over the new name?

Predict

After renaming weight to weight_kg, what does SUM(weight_kg) over both files return?

Both files' weightsOnly the new file'sAn error

Run it

1Install uv once. It fetches Python and the lab's packages for you.
Terminal
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2Get the script. It is the whole lab: read it, then download it.

rename-is-drop-plus-add-in.py38 linesDownload

View the code

rename-is-drop-plus-add-in.py

# /// script
# requires-python = ">=3.10"
# dependencies = ["duckdb"]
# ///
# Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
# https://thedataengineering.com/lab-code/LICENSE.txt
"""Rename is drop plus add, in a table.

Two Parquet files, an older one with a column weight and a newer one where it
is called weight_kg, read together by DuckDB with union_by_name; then the
read-side workaround COALESCE(weight_kg, weight).
"""
import argparse, json, os, tempfile
import duckdb

p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--json", help="Also write the results to this JSON file.")
args = p.parse_args()

d = tempfile.mkdtemp(); c = duckdb.connect()
c.execute(f"COPY (SELECT 'S1' AS id, 12.5 AS weight) TO '{d}/old.parquet' (FORMAT parquet)")
c.execute(f"COPY (SELECT 'S2' AS id, 3.2 AS weight_kg) TO '{d}/new.parquet' (FORMAT parquet)")
files = f"['{d}/old.parquet', '{d}/new.parquet']"
union = c.execute(f"SELECT id, weight, weight_kg FROM read_parquet({files}, union_by_name = true) ORDER BY id").fetchall()
cols = [r[0] for r in c.execute(f"DESCRIBE SELECT * FROM read_parquet({files}, union_by_name = true)").fetchall()]
fixed = c.execute(f"SELECT id, COALESCE(weight_kg, weight) AS weight_kg FROM read_parquet({files}, union_by_name = true) ORDER BY id").fetchall()
total_naive = c.execute(f"SELECT sum(weight_kg) FROM read_parquet({files}, union_by_name = true)").fetchone()[0]
rows = [{"id": i, "weight": w, "weight_kg": wk, "coalesced": f} for (i, w, wk), (_, f) in zip(union, fixed)]
summary = {"columns": ", ".join(cols), "sum_weight_kg_naive": float(total_naive), "sum_coalesced": float(sum(f for _, f in fixed))}
print("columns:", summary["columns"])
for r in rows: print(f"{r['id']}: weight {r['weight']}, weight_kg {r['weight_kg']}, COALESCE(weight_kg, weight) {r['coalesced']}")
print(f"sum(weight_kg) without the workaround: {total_naive}; with it: {summary['sum_coalesced']}")
for f in os.listdir(d): os.remove(os.path.join(d, f))

if args.json:
    with open(args.json, "w") as f:
        json.dump({"summary": summary, "tables": {"rows": rows}}, f, default=float)

3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal
```
uv run rename-is-drop-plus-add-in.py
uv run rename-is-drop-plus-add-in.py --help     # every knob
```

Our run

Two files read by name

Two files read by name
id	weight	weight_kg	COALESCE(weight_kg, weight)
S1	12.5	—	12.5
S2	—	3.2	3.2

Reference runApple M2 · 8 cores · 8 GB

Output · uv run rename-is-drop-plus-add-in.py

columns: id, weight, weight_kg
S1: weight 12.5, weight_kg None, COALESCE(weight_kg, weight) 12.5
S2: weight None, weight_kg 3.2, COALESCE(weight_kg, weight) 3.2
sum(weight_kg) without the workaround: 3.2; with it: 15.7

macOS 26.6.2 · Python 3.13.3 · duckdb 1.5.6 · run 2026-10-20

What we found

The read returned the columns id, weight, weight_kg, each null for the other file's row. A sum over weight_kg gave 3.2, silently leaving out the old row; with COALESCE(weight_kg, weight) it gave 15.7.

Read it

Every reader that uses the new name must also know the old one until the old data is gone, which is why a rename is the most expensive of the "simple" schema changes. The coalesce is a workaround in each query; expand-and-contract puts it in one place, at write time.