Monotonic clocks, wall clocks and replay determinism

Which clock measures a duration, and why must a run's logical time be an input, never an ambient read?

A duration must be measured with a monotonic clock, never by subtracting two wall-clock readings: a wall clock can step backwards or forwards by any amount (an NTP correction, a VM migration, a manual change), so a timeout computed from it can last for hours or a lease can look valid after it expired (clocks: three requirements). That was Kavya's third symptom. Stored timestamps and comparisons across machines use the wall clock; durations, timeouts, backoff and rate limiters use the monotonic one.

import time

# Right: measure a duration with the monotonic clock
start = time.monotonic()
do_work()
elapsed = time.monotonic() - start

# Wrong for durations: time.time() is the wall clock and can jump
# deadline = time.time() + 30   # a backwards step makes this wait for hours

Replay determinism

The same issue appears in a pipeline as replay determinism: a computation that reads the ambient clock cannot be replayed, and without replay there is no backfill, no regression test and no way to reproduce an incident. It is the correctness rule of this mechanism (output must not depend on how fast or when the pipeline ran) at its most concrete. The other sources of non-determinism are fault-tolerance's; for time, one rule fixes them:

# Bad: result depends on when the job runs
def daily_report():
    today = date.today()
    return query(f"... WHERE event_date = '{today}'")

# Good: the data interval is passed in, so any day can be re-run identically
def daily_report(run_date: date):
    return query(f"... WHERE event_date = '{run_date}'")

This is why mature job schedulers parameterize every run on a data interval (for example "the day 2025-03-05"). The deeper rule: hidden state means non-reproducibility under replay.