CI-time schema diff

Why is catching a breaking change at merge the highest-leverage check?

CI (continuous integration) is the automatic build and test run that happens when code is proposed for merging. A schema diff in CI compares the schema your code produces with the registered schema.

There are three places to detect a breaking change, with roughly an order of magnitude in cost between each:

The mechanism is unremarkable:

1. derive the schema your code produces,
2. diff it against the registered one,
3. classify the diff using the operation table,
4. fail the build when the class is breaking.

def classify(old, new):          # old/new: {field: {"type":..., "required":bool, "default":...}}
    problems = []
    for f, o in old.items():
        if f not in new:
            if o["required"]:
                problems.append(f"drop of required field {f}")
        elif new[f]["type"] != o["type"]:
            problems.append(f"type change on {f}: {o['type']} -> {new[f]['type']}")
    for f, n in new.items():
        if f not in old and n["required"] and n.get("default") is None:
            problems.append(f"new required field {f} without default")
    return problems              # non-empty -> fail the build

The precondition is that the schema must be derivable from the code. This is the real argument for a schema generated as an artifact in the repository over a schema produced as a runtime side effect.

This is the highest-leverage mechanism here, and it is missing from many lists because it is not a data mechanism at all. It is the "fail early" idea applied at the earliest point where the change exists only in one person's branch and has produced zero bytes of data. (Note the difference: fail fast means failing as soon as a problem happens at runtime; fail early means failing before anything runs.)

Detect at merge, block at deploy, discover at runtime: each step later costs about ten times more.