Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Evaluation splits and pinned data

When does a held-out score mean what you think, and how do you split and pin data so it does?

A model's held-out score is a measurement like any other: a number that means what you think only if the test set is independent of the training set and the data can be reconstructed. The model's oracle (the thing that could say "this output is right") is outside the system, so correctness cannot be asserted; what remains is reproducibility, and here it means more than usual: the inputs must be reconstructible, not only the code.

Pin the data, not the query

Pinning a query is not enough, because the tables underneath it change. The mechanisms are the ones data systems already offer:

  • immutable snapshots, table versions and tags;
  • time travel (reading a table as it was at an earlier point or version);
  • content hashing of the resulting dataset.

A training run references a snapshot id, not a table name.

This conflicts with retention: a system that degrades or deletes old data cannot reproduce a result computed on it. The usual resolution is to keep evaluation datasets outside the retention policy entirely.

Splits: the mechanism most often done wrong

A split divides data into a training set (to fit the model) and a test set (to measure it). A random split assumes the rows are exchangeable, meaning any row could equally well be in either set. That is the independence assumption stated precisely, and it is rarely true. For many real datasets it is false:

  • Temporal data. A random split puts future rows in training and past rows in test. That is the same future leakage as joining on the current version, in different clothes. The correct split is temporal: train on before, test on after.
  • Grouped data. With many rows per customer, a random split puts the same customer on both sides, and the model memorizes the customer instead of learning the pattern. The correct split is by group: all rows of a customer go to the same side.

The general rule: a split must respect every dependency structure in the data. The default assumption of independence is wrong exactly where the data is most interesting. The violation has no symptom except an over-optimistic number.

Stable splits from a hash

The split must also be stable across reruns. If you use a random shuffle, adding new rows reshuffles everything, and yesterday's test set becomes today's training set. Instead, derive the side from a hash of a stable key:

import hashlib

def side(customer_id: str) -> str:
    # md5 is used for stable bucketing, not for security.
    # (Python's built-in hash() is randomized per process, so do not use it here.)
    bucket = int(hashlib.md5(customer_id.encode()).hexdigest(), 16) % 100
    return "train" if bucket < 80 else "test"   # 80% / 20%, grouped by customer

The same customer always lands on the same side, whenever and wherever the code runs. This is deterministic partitioning (hash a key to choose a bucket) used for reproducibility instead of for spreading load: a small but nice case of one mechanism serving a very different purpose.

One line: pin the data version, and derive the split from a hash of a key that respects the data's real dependency structure. that temporal and grouped data do not have.