Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Versioning, branches and tags

Why is branching data free while merging it is hard?

storage plus a pointer is a version. Named pointers are tags. Several movable pointers over shared history are branches. The cost structure is interesting, and it is asymmetric in a way that the analogy with git (the source-control tool) hides.

  • , because nothing is copied. You clone a pointer. Reading a branch is free.
  • Merging is not O(1), and it is generally undecidable without knowledge of the data's meaning. File-level merging has no rule that preserves meaning. Example: two branches both appended 1,000 rows to the same table. Are those the same 1,000 corrected rows, or 2,000 different rows? The system cannot tell, so a union may double-count.

So what "git for data" really delivers is: cheap branch, cheap read, cheap rollback, and (a merge where one branch has not changed since the split, so the other can simply be adopted). Three-way merge with conflict resolution is the part of the analogy that does not transfer, and it is the part the name promises.

What it does deliver is a lot:

  • Write-audit-publish becomes natural: write new data on a branch, check it, then publish by moving the main pointer. Readers never see unchecked data.
  • Reproducibility of experiments is free: pin the version you trained on.
  • Time-travel reads ("show the table as of last Tuesday") are the same mechanism, read differently.
  • Rollback is a pointer swap rather than a restore from backup: recovery at O(1) instead of O(data).

This is the copy-on-write mechanism for the third time in a data system: first for durability, second as a maintenance debt (old files must be cleaned), and here as a version-control primitive. Same mechanism, same tax each time: space plus a collector.

The binding constraint is retention. Depth of is a storage bill and a direct limit on how aggressively old files can be deleted (). Worked example: a 10 TB table in which each day's updates rewrite 2% of the files (200 GB). Keeping 30 days of snapshots keeps about 30 × 200 GB = 6 TB of extra files (60% more storage). How far back you can read is an economic choice, not a technical capability, and expiring snapshots is where it is paid.

Immutable history also collides with deletion duties (for example, a legal duty to delete a person's data). The resolutions are crypto-shredding (encrypt each person's data with its own key and delete the key, so the old copies become unreadable) and deletion vectors (small side files that mark rows as deleted so readers skip them). Both make data unreadable without making it absent.

Branching data is free because nothing is copied. Merging data is hard for the same reason git merges text and not meaning.