Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Schema validation and the per-value tier

Why is type checking free, and why can per-value checks verify form but never existence?

Schema validation, and why type checking is free

Schema evolution is about how structures change compatibly over time. Schema , the topic here, is the runtime act of deciding whether a record conforms at a boundary. (What happens to a record that does not is a schema-evolution decision.) Here the question is what such a check can prove.

One fact dominates: You are already touching the bytes to produce a value. Asserting that the bytes form a valid 32-bit integer is the same operation as producing the integer. So the whole per-value tier of checks (type, not-null, range, , regex) collapses into the parse step at almost zero extra cost, and every serious ingestion path folds them together.

The ceiling: form versus existence

The consequence is a hard ceiling. Per-value checks can verify well-formedness and nothing else.

  • A regular expression confirms that an email looks like an email. It cannot confirm that the mailbox exists.
  • A (a checksum built into an identifier, as in many account or card numbers) confirms the number is internally consistent. It cannot confirm that the account exists.

Existence needs the external authority, which means a cross-dataset check (a ) or a . Most quality programmes stall exactly here: they exhaust the free tier, report high coverage, and have verified only syntax. That is Ravi's second symptom: 99999999 is a perfectly formed integer.

Two per-value cases that need their own treatment

Null is the most overloaded value in data engineering. It can mean: absent (no value was given), not applicable (the field does not apply), unknown (a value exists but we do not have it), and "the join found nothing". A null-rate check cannot tell these meanings apart. So a null-rate alarm means different things depending on which sense dominates, and the mix changes when upstream logic changes.

The solution that teams converge on is to move the distinction out of nullity and into the value domain: explicit marker values such as 'NOT_APPLICABLE' or 'UNKNOWN', or explicit sentinel members. Once the distinction is a real value, it can be checked by a set-membership test. It has dropped back into the free tier. This is the same move as the above: make the anomaly an explicit value so that it can be counted.

-- Instead of one NULL with four meanings, use explicit states.
CREATE TABLE orders (
  order_id       BIGINT,
  coupon_code    VARCHAR,       -- real code, or one of the markers below
  -- coupon_code values: 'NONE_USED', 'NOT_APPLICABLE_PARTNER_ORDER', 'UNKNOWN_LEGACY'
  delivery_notes VARCHAR
);
-- A set-membership check now works and counts each meaning separately:
SELECT coupon_code, COUNT(*) FROM orders
WHERE coupon_code IN ('NONE_USED','NOT_APPLICABLE_PARTNER_ORDER','UNKNOWN_LEGACY')
GROUP BY coupon_code;

Set membership (the value must be one of an allowed list) looks like a per-value check. It is secretly a cross-dataset check. The list of valid values is itself data. It has a maintainer and a release schedule. When the business adds a new order-status code, the check fires, and the data is right while the check is wrong. This is the three-meanings ambiguity in its simplest form. It is also the principle stated in Propositions about the world: the expectation is an artifact that ages.

Compression: decoding pays for the whole per-value tier, and that is also its ceiling. Form is local. Existence is not.