Schema inference and drift

Why is an inferred schema non-deterministic, and how can inference become a drift detector instead?

Inference means reconstructing a schema from a sample of the data, instead of reading a declared one. Many tools do this: they read the first few thousand rows of a JSON or CSV file and decide the column types. Every failure of inference traces back to one cause: the sample is not the population.

Type instability

Imagine a column that is an integer for ten thousand rows and a string in row ten thousand and one. What type does the tool infer? It depends on:

• how deep it sampled,
• which file it read first,
• how large the file was that day.

So the schema is non-deterministic across runs over identical data. The same data can produce different schemas on different days. That is Sara's postal_code problem: if the tool samples the first 10,000 rows, it says integer. If a merchant's text value falls inside the sample, it says text.

Structural drift

• A new field appears in the data. The inferred schema silently widens. Every downstream SELECT * consumer now receives a different shape of table, with no deploy and no review.
• A field stops appearing for a while. The column vanishes from the inferred schema. Queries that name it fail, or worse, resolve to something else.

Inference versus statistics

Compare with the statistics a database planner keeps. A planner samples a table to estimate how many rows match a filter. These statistics are also sampled and also stale. But wrong statistics cost a bad plan: the query runs slower, and the system is built to tolerate an estimate being wrong. Wrong inference costs wrong data. Sampling is legitimate for cost estimation and illegitimate for semantics, because an estimate is allowed to be wrong, and a type is not.

The useful inversion

The useful move is to change the role of inference:

1. Infer once.
2. Freeze the result as a declared artifact (a schema file in version control).
3. Treat every later difference found by inference as an alert, not an update.

That changes inference from a binding mechanism, where it is dangerous, into a drift detector, where it is the cheapest one available.

import json

frozen = {"shipment_id": "string", "postal_code": "string", "weight_kg": "double"}

def infer(records):
    out = {}
    for r in records:
        for k, v in r.items():
            t = {int: "long", float: "double", str: "string"}[type(v)]
            out.setdefault(k, set()).add(t)
    return out

def drift(frozen, inferred):
    alerts = []
    for k, types in inferred.items():
        if k not in frozen:
            alerts.append(f"new field {k}")
        elif types != {frozen[k]}:
            alerts.append(f"type change in {k}: {sorted(types)} vs {frozen[k]}")
    for k in frozen:
        if k not in inferred:
            alerts.append(f"field {k} missing from sample")
    return alerts

Inference makes your schema a non-deterministic function of your sample. Freeze the first one; use later ones as alarms.