Fixtures and data diff: validating the code instead of the data

How do fixtures and data diffs validate a code change rather than a dataset?

Everything so far checks a dataset. These two check a change, and they catch different things.

A golden fixture is a frozen input and an expected output. It is deterministic, cheap, runs in continuous integration, and catches logic regressions exactly. It is blind to everything about production reality: new value distributions, new edge cases, volume, skew. Its maintenance cost grows with schema changes. And it has one typical corruption: when a fixture fails, the fastest fix is to regenerate the expected output from the code under test. Do that once, and the test only asserts that the code is deterministic. A meaningful share of golden tests in practice are in this state, and they pass forever.

Data diff is the differential applied to code. Run the old and the new version of the code over the same input snapshot and reconcile the two outputs against each other. It needs no expectations at all. You do not predict what your change will break. You are shown everything it changed, and you decide.

-- Data diff: old pipeline output vs new pipeline output on the same input snapshot.
SELECT 'only_in_old' AS side, o.* FROM out_old o
  LEFT JOIN out_new n USING (order_id) WHERE n.order_id IS NULL
UNION ALL
SELECT 'only_in_new', n.* FROM out_new n
  LEFT JOIN out_old o USING (order_id) WHERE o.order_id IS NULL
UNION ALL
SELECT 'changed', n.* FROM out_new n JOIN out_old o USING (order_id)
  WHERE o.total <> n.total OR o.status <> n.status;

That inversion is the important part. Every other mechanism here tests a hypothesis someone imagined, so coverage is limited by imagination. A diff enumerates the change set. Within its scope, recall is complete by construction, not by diligence. It is the only mechanism here with that property.

The scope is the boundary condition, and it is narrow. A diff tells you what your change altered. It is silent about defects present in both versions, and it says nothing about whether the new output is right, only that it differs. Its costs are 2× compute on a representative input, plus a stable snapshot to run both against. Its practical failure is that a legitimate change to a widely used column produces a diff with millions of changed rows, which nobody can review. The escape is to summarise the diff by column and by size of change instead of by row, which brings you back to the reconciliation ladder: a diff is readable only if it is shown at the right rung.

Compression: fixtures test logic against a frozen world. Diffs test a change against the real one, and are the only mechanism here with complete recall over their own scope.