Training-serving skew: one number computed twice
Why can two implementations of the same feature never be verified against each other, and what does a feature store really remove?
A model feature (an input value such as "this customer's average order value over the last 30 days") is usually computed twice: by a batch job over history, to build the training set, and by an online service at request time, to score one customer now. Two systems, two schedules, often two languages, and nothing in the architecture forces them to agree. Other duplications have a common substrate to check against: a materialized view can be validated against its base table, a replica checksummed against its leader. Here one side is a batch job over a lake and the other a service reading a key-value store, so divergence is undetectable by construction, and the model does not raise an error on inputs from a distribution it never saw.
Training-serving skew means the model sees different feature values in production than it saw in training. It is not one failure but four, with different fixes. "Use a feature store" is offered as the fix for all four, and it only addresses two or three of them.
| Kind of skew | What happens | Can shared code fix it? |
|---|---|---|
| Logic skew | The batch and online implementations of the same feature drift apart. Someone changes the offline SQL and not the online service. | Yes: one specification, two execution modes. |
| Timing skew | Same logic, but offline sees a complete day and online sees a partly arrived one. | No. The difference is in what has arrived, not in what is computed. Point-in-time joins on availability time do the work. |
| Data-source skew | Offline reads a warehouse table, online reads an operational store. Different pipelines, different late-data handling. | Only by making one the source of the other. |
| Distribution skew | The world moved. Nothing is wrong with either implementation. | No. Detectable only by monitoring (drift monitoring), never preventable. |
What a feature store is
The core idea is small: define the feature once, then materialize it to an offline store for training and to an online store for serving, from the same computation.
- The offline store is optimized for large point-in-time-correct scans.
- The online store is optimized for single-key, low-latency reads (for example, "give me the features of customer 7 in 5 ms").
It is one logical dataset materialized into two physical layouts with opposite access patterns, the same idea as a materialized view with a different index than its base table, or a search index beside its document store. Here the two copies are not just different indexes but different systems, with independent write paths, independent failure modes and independent freshness.
Why the guarantee is weaker than it is usually claimed
Dual materialization removes logic skew by construction. It does not remove the possibility that the two stores are at different points in time: one pipeline succeeded and the other failed, or online is streaming while offline is nightly. Sharing the definition does not share the execution. The residual failure is about time, not logic.
The stronger version is: compute once, in the streaming path, and write the same computed value to both stores. Then the serving value is the source of the training value instead of a parallel computation of it. This is the only arrangement that makes skew structurally impossible instead of merely unlikely. It costs you the ability to define features in batch SQL, which is most of the reason people do not do it.
The honest summary: a feature store converts an undetectable correctness problem into a visible freshness problem. That is a real improvement, because freshness can be monitored and alerted on. It is also much less than "skew solved".
One line: shared definition removes logic skew; only shared execution removes timing skew; everything else becomes a freshness metric that you now have to watch.
The general rule
Examples: training and serving features; a dual write to two stores; a metric defined in the warehouse and again in a BI (business intelligence) tool; a reimplemented pipeline running beside the original during a migration. In every case the only mechanisms available are:
- a shared definition (removes logic divergence, not timing divergence);
- a shared execution (removes both, costs flexibility);
- continuous reconciliation against sampled outputs (detects, but does not prevent).
Reconciliation jobs exist because this rule is true. Training-serving skew is where the consequence is most severe, because the consumer, a model, does not raise an error: it produces a confident number.