A layer with no oracle

Why is data modeling a different kind of problem from the storage and execution problems below it, and which four ideas does it reduce to?

The system works, and lies

Every layer below data modeling has a machine-checkable oracle, a way for the machine itself to tell you that it is wrong. A checksum catches a corrupted disk block. A serializability test catches a lost update. The cost of a query plan can be measured in seconds and dollars. A watermark (the system's estimate of "all events before time T have arrived") is either past an event or not. When those layers are wrong, something breaks and you find out.

Data modeling has no oracle. A star schema built at the wrong grain runs perfectly. It returns in 200 ms. It passes every constraint check. And it reports revenue that is 14% too high, forever.

That is the categorical difference, and it drives everything else here. The machine cannot validate the model. So validation has to come from people agreeing in advance on what a row means. And agreement is expensive in a way that bytes are not.

The constraint that stops mattering: physical access cost

Classical teaching says: normalize your tables (split them so each fact is stored once, explained in Normalization and denormalization). That advice was rational when storage cost dominated and every join meant a random disk trip. Today four mechanisms have absorbed most of that argument, for and against both normalization and denormalization:

• columnar layout (each column stored separately, so unused columns are not read);
• lightweight compression (dictionary encoding and run-length encoding make repeated values nearly free);
• zone maps and pruning (small statistics per file that let the engine skip files that cannot match);
• broadcast joins and runtime filters (joins to small tables are cheap).

So the old folklore, "normalize to save space, denormalize for speed", is now a weak and often wrong justification. The real reasons to choose a table shape are semantic (what does a row mean?) and organizational (who owns this definition?). Treat any modeling argument that is purely about I/O as a claim to check first. Usually the storage and execution layers have already answered it.

The constraint that appears: coordination cost

The new cost in this mechanism is coordination cost: it scales with the number of parties who must agree, not with the volume of data. Ten thousand rows read by forty teams is a harder modeling problem than ten billion rows read by one team. This is the first layer across the platform where N is people.

The governing question of the layer is:

What does one row mean, who is allowed to change that answer, and what breaks downstream when they do?

---

The four primitives

Almost everything in a data modeling course reduces to four ideas.

Grain is what exactly one row is. It is the atomic act of modeling. Normalization, star versus snowflake, the types of fact tables, One Big Table and nested modeling are all choices of grain, or strategies for mixing grains safely.

Identity is when two references are the same thing. Surrogate versus natural keys, conformed dimensions and entity resolution (deciding that two records describe the same person) are all this one question at different confidence levels. The identity is asserted by decree, or asserted by agreement, or inferred by probability. Identity is never discovered. It is always declared, and the declaration cannot be proven false from inside the system.

Time is which version of an entity a fact joins to. Slowly changing dimensions, event sourcing and activity schema all answer it. Systems below this mechanism give you the mechanics: clocks, versioned reads, "show me the data as of time T". This mechanism asks only which semantics the business definition requires, and what it costs to be wrong. Keep the boundary sharp. "How do I read as of T" is a mechanics question. "As of which T should this number be computed, and who says so" is a modeling question.

Boundary is where one model ends and another begins. The bus matrix, medallion layering, the semantic layer and data contracts are boundary mechanisms. They are organizational mechanisms wearing schema costumes.

Modeling styles are positions on three axes

The named modeling styles are not competing religions. They are positions on three choices:

1. Early versus late binding (decide structure at load time or at read time).
2. Precompute versus compute on read (do the work at write time or at query time).
3. Who pays for the join: the modeler once, or every consumer forever.

That is the whole argument, and it can be decided per number of consumers, not by taste. A style that pushes the join onto readers is fine for two readers and ruinous for two hundred.