Skip to content
About
Main content

Replay determinism check

Measure your machineNeeds Python

Problem

is the cheapest recovery: rerun the job over the same input and get the same output. It works only if the job is deterministic, and hidden non-determinism is easy to add: reading the clock instead of the event's own time, looking up a table that changes between runs, iterating an unordered collection. None of these fail a test; they make a rerun quietly produce different numbers.

Find hidden non-determinism before you need replay. Take a small aggregation function that reads events and writes daily totals. Add a call to datetime.now() and a lookup of a mutable dictionary (change one value between runs). Run the job twice over the same input and compare the output hashes. Then fix it: pass the event time in the event, and version the lookup table by date. Run twice again.

The question. Do two runs over the same input produce the same output, before and after the fix?


Predict

Run twice over the same input with now() and a mutable lookup: do the two outputs match?

Run it

  1. 1Install uv once. It fetches Python and the lab's packages for you.

    Terminal
    curl -LsSf https://astral.sh/uv/install.sh | sh
  2. 2Get the script. It is the whole lab: read it, then download it.

    replay-determinism-check.py52 linesDownload
    View the code
    replay-determinism-check.py
    # /// script
    # requires-python = ">=3.10"
    # ///
    # Copyright (c) 2026 Aayush Sharma · SPDX-License-Identifier: MIT
    # https://thedataengineering.com/lab-code/LICENSE.txt
    """Replay determinism check.
    
    A daily-totals job over a fixed event file that calls datetime.now() and
    reads a mutable lookup table; run twice and compare output hashes. Then the
    fixed version: the event time comes from the event, and the lookup is
    versioned by date.
    """
    import argparse, datetime, hashlib, json, random
    
    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    p.add_argument("--events", type=int, default=50_000, help="Events.")
    p.add_argument("--seed", type=int, default=0, help="Random seed.")
    p.add_argument("--json", help="Also write the results to this JSON file.")
    args = p.parse_args()
    random.seed(args.seed)
    events = [{"ts": f"2025-10-{random.randint(1, 30):02d}T{random.randint(0, 23):02d}:00", "sku": random.choice("ABCD"), "qty": random.randint(1, 5)}
              for _ in range(args.events)]
    prices = {"A": 10, "B": 20, "C": 30, "D": 40}                     # a mutable lookup someone edits
    price_versions = {"2025-10-01": {"A": 10, "B": 20, "C": 30, "D": 40}}
    
    def job_v1(run_time):
        totals = {}
        for e in events:
            day = run_time.date().isoformat()                         # bug: processing time instead of event time
            totals[day] = totals.get(day, 0) + e["qty"] * prices[e["sku"]]
        return hashlib.sha256(json.dumps(totals, sort_keys=True).encode()).hexdigest()[:16]
    
    def job_v2():
        totals = {}
        for e in events:
            day = e["ts"][:10]
            table = price_versions[max(k for k in price_versions if k <= day)]       # the version valid on the event's date
            totals[day] = totals.get(day, 0) + e["qty"] * table[e["sku"]]
        return hashlib.sha256(json.dumps(totals, sort_keys=True).encode()).hexdigest()[:16]
    
    a = job_v1(datetime.datetime(2025, 11, 1, 9)); prices["C"] = 35
    b = job_v1(datetime.datetime(2025, 11, 2, 9))
    c = job_v2(); price_versions["2025-11-02"] = {"A": 10, "B": 20, "C": 35, "D": 40}
    d = job_v2()
    rows = [{"version": "now() + mutable lookup", "run1": a, "run2": b, "identical": "yes" if a == b else "no"},
            {"version": "event time + versioned lookup", "run1": c, "run2": d, "identical": "yes" if c == d else "no"}]
    for r in rows: print(f"{r['version']:<32} run 1 {r['run1']}  run 2 {r['run2']}  identical: {r['identical']}")
    
    if args.json:
        with open(args.json, "w") as f:
            json.dump({"tables": {"versions": rows}}, f)
    
  3. 3Run it in the folder you saved it to, with the defaults first. Then change a knob from the table below and run it again.

    Terminal
    uv run replay-determinism-check.py
    uv run replay-determinism-check.py --help     # every knob
Knobs
Knob · defaultWhat it changes
--events50,000 Events.
--seed0 Random seed.

Our run

Two runs over the same 50,000 events

Two runs over the same 50,000 events
VersionRun 1 hashRun 2 hashIdentical
now() + mutable lookup121032cdfd34cbebb1858511772ef68bno
event time + versioned lookupf3db8463513aa095f3db8463513aa095yes
Reference runApple M2 · 8 cores · 8 GB
Output · uv run replay-determinism-check.py
now() + mutable lookup           run 1 121032cdfd34cbeb  run 2 b1858511772ef68b  identical: no
event time + versioned lookup    run 1 f3db8463513aa095  run 2 f3db8463513aa095  identical: yes
macOS 26.6.2 · Python 3.13.3 · run 2026-10-20

What we found

With now() and the mutable lookup, the two runs disagreed (identical: no). With the event time and a lookup versioned by date, they agreed (yes).


Read it

This is a cheap test you can add to a build pipeline: run the job twice over a fixed input and compare hashes. It finds the non-determinism while it is a code change, not during an incident when the replay you were counting on produces different numbers from the original.