Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Medallion layering

Why materialize a pipeline at the boundaries where trust changes, and what makes a layer a contract rather than a copy?

Medallion layering (often called bronze, silver and gold layers) organizes a pipeline into stages: raw data, cleaned data and business-ready data.

The constraint: a pipeline mixes transformations of different reversibility and different trust.

  • Parsing is mechanical.
  • Deduplication is lossy: it throws rows away.
  • Business logic is opinionated and will be wrong at some point.

If all three happen in one step, fixing the third requires re-obtaining the input of the first, which may no longer exist. That is Nadia's situation.

So you materialize at the boundaries where trust changes, and each boundary is a : a stored state from which you can recompute everything after it. That is the whole mechanism. It is the same idea as checkpoint recovery in stream processors (save state so that you restart from the checkpoint, not from the beginning) and as breaking a pipeline by writing intermediate results to storage instead of streaming everything through memory. The choice of where to checkpoint is the same trade in all three:

Three corrections

Three layers is arbitrary. The principle is "materialize where replay boundaries and trust levels change." That might be two layers or five. The number is a convention that got a name.

The layers are a contract structure only if each boundary publishes guarantees: schema, freshness, uniqueness, null policy, and the meaning of the . Without published guarantees, three layers is three copies of the data and a longer pipeline. This is where most implementations fail, and they fail for organizational reasons, not technical ones: publishing a guarantee means accepting an obligation, and nobody was assigned one.

Raw is a replay point only if the source is not itself replayable. If your upstream is an immutable log with enough retention (a message log that keeps every message for months), the raw layer is a redundant copy whose justification is retention depth and decoupling, not correctness. Say which one you are buying.

The cost

: roughly 3× storage, latency equal to the sum of the hops, and a lineage graph with three times as many nodes. Worked example: 10 TB of raw data becomes about 10 + 8 + 4 = 22 to 30 TB over three layers, depending on how much each layer shrinks it. If each hop takes 10, 15 and 20 minutes, the end-to-end delay is 45 minutes, even if every step is fast on its own. (These numbers are illustrations.)