Replay determinism check
Measure your machineNeeds Python
Problem
Replay is the cheapest recovery: rerun the job over the same input and get the same output. It works only if the job is deterministic, and hidden non-determinism is easy to add: reading the clock instead of the event's own time, looking up a table that changes between runs, iterating an unordered collection. None of these fail a test; they make a rerun quietly produce different numbers.
Find hidden non-determinism before you need replay. Take a small aggregation function that reads events and writes daily totals. Add a call to datetime.now() and a lookup of a mutable dictionary (change one value between runs). Run the job twice over the same input and compare the output hashes. Then fix it: pass the event time in the event, and version the lookup table by date. Run twice again.
The question. Do two runs over the same input produce the same output, before and after the fix?
Predict
The first pair of outputs differs (hashes do not match). After the fix, the hashes are identical.
Run it
1Install uv once. It fetches Python and the lab's packages for you.
Terminal curl -LsSf https://astral.sh/uv/install.sh | sh2Get the script. It is the whole lab: read it, then download it.
View the codeHide the code
replay-determinism-check.py # /// script # requires-python = ">=3.10" # /// # Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT # https://thedataengineering.com/lab-code/LICENSE.txt """Replay determinism check. A daily-totals job over a fixed event file that calls datetime.now() and reads a mutable lookup table; run twice and compare output hashes. Then the fixed version: the event time comes from the event, and the lookup is versioned by date. """ import argparse, datetime, hashlib, json, random p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) p.add_argument("--events", type=int, default=50_000, help="Events.") p.add_argument("--seed", type=int, default=0, help="Random seed.") p.add_argument("--json", help="Also write the results to this JSON file.") args = p.parse_args() random.seed(args.seed) events = [{"ts": f"2025-10-{random.randint(1, 30):02d}T{random.randint(0, 23):02d}:00", "sku": random.choice("ABCD"), "qty": random.randint(1, 5)} for _ in range(args.events)] prices = {"A": 10, "B": 20, "C": 30, "D": 40} # a mutable lookup someone edits price_versions = {"2025-10-01": {"A": 10, "B": 20, "C": 30, "D": 40}} def job_v1(run_time): totals = {} for e in events: day = run_time.date().isoformat() # bug: processing time instead of event time totals[day] = totals.get(day, 0) + e["qty"] * prices[e["sku"]] return hashlib.sha256(json.dumps(totals, sort_keys=True).encode()).hexdigest()[:16] def job_v2(): totals = {} for e in events: day = e["ts"][:10] table = price_versions[max(k for k in price_versions if k <= day)] # the version valid on the event's date totals[day] = totals.get(day, 0) + e["qty"] * table[e["sku"]] return hashlib.sha256(json.dumps(totals, sort_keys=True).encode()).hexdigest()[:16] a = job_v1(datetime.datetime(2025, 11, 1, 9)); prices["C"] = 35 b = job_v1(datetime.datetime(2025, 11, 2, 9)) c = job_v2(); price_versions["2025-11-02"] = {"A": 10, "B": 20, "C": 35, "D": 40} d = job_v2() rows = [{"version": "now() + mutable lookup", "run1": a, "run2": b, "identical": "yes" if a == b else "no"}, {"version": "event time + versioned lookup", "run1": c, "run2": d, "identical": "yes" if c == d else "no"}] for r in rows: print(f"{r['version']:<32} run 1 {r['run1']} run 2 {r['run2']} identical: {r['identical']}") if args.json: with open(args.json, "w") as f: json.dump({"tables": {"versions": rows}}, f)3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.
Terminal uv run replay-determinism-check.py uv run replay-determinism-check.py --help # every knob
| Knob · default | What it changes |
|---|---|
| --events50,000 | Events. |
| --seed0 | Random seed. |
Our run
Two runs over the same 50,000 events
| Version | Run 1 hash | Run 2 hash | Identical |
|---|---|---|---|
| now() + mutable lookup | 121032cdfd34cbeb | b1858511772ef68b | no |
| event time + versioned lookup | f3db8463513aa095 | f3db8463513aa095 | yes |
now() + mutable lookup run 1 121032cdfd34cbeb run 2 b1858511772ef68b identical: no
event time + versioned lookup run 1 f3db8463513aa095 run 2 f3db8463513aa095 identical: yesWhat we found
With now() and the mutable lookup, the two runs disagreed (identical: no). With the event time and a lookup versioned by date, they agreed (yes).
Read it
This is a cheap test you can add to a build pipeline: run the job twice over a fixed input and compare hashes. It finds the non-determinism while it is a code change, not during an incident when the replay you were counting on produces different numbers from the original.