Variant and union columns

When is a variant column the right escape hatch, and what keeps it from eating the schema?

Variant, union, and the licensed escape hatch

Some payloads really are open: per-tenant custom fields, webhooks from third parties, device telemetry that changes with the firmware. Forcing them into a closed schema turns every customer onboarding into a deploy. That is an organizational cost so high that people route around it with a stringified JSON blob in a text column. The blob is the same thing as a variant column, with worse tooling.

The mechanism

A variant column is a column whose type is "any self-describing value": the value carries its own structure, like a JSON document. Usually the system also shreds frequently present paths into real sub-columns. Shredding means that if 90% of records have custom.bin_number, the engine stores that path as its own typed column behind the scenes. This partly recovers scan speed.

A union type is the closed cousin: a field that may be one of an enumerated set of alternatives (for example, integer or string). Because the set is finite, the system can check it. A variant is open, so it cannot be checked.

What a variant buys, and what kills it

What a variant buys is scoped late binding. The escape hatch is bounded to one column instead of contaminating the whole record.

What kills it is scope creep. Writing into the variant is always the locally cheapest action. Without a rule against it, everything ends up there, and you have schema-on-read with extra steps.

The rule against it is a promotion path: a path that is present in more than X% of records for more than Y days is promoted to a declared column. Without a promotion policy, the escape hatch becomes the design.

A variant is a licensed, scoped region of late binding; it needs a promotion path or it eats the schema.

Case, resolvedConstructed case · Sara at Parcelo

Two dashboards, two numbers for the same week

Two dashboards over raw JSON disagreed on the same metric, an inferred postal_code flipped type and broke a nightly join, and a merchant wanted to send its own custom fields. Where it started: Where structure is bound →

Two dashboards that disagree. Parcelo defines a required core (shipment_id, event_type, occurred_at, tenant_id, schema_version), declared and checked at write time. "Delivered" is now defined once, in the core. Both dashboards read the same typed event_type and occurred_at. The 41,200 versus 40,870 gap was caused by the two private date parsers and disappears.
The type flip. The team freezes the inferred schema in the repository and runs inference only as a nightly check. When postal_code appears as text, the check raises an alert instead of silently changing the type (and postal_code is declared as string, overriding the inference).
Merchant custom fields. They go into a variant column custom, with the promotion rule above.