Hidden state breaks replay
A replayable pipeline is a pure function of its retained inputs.
What it is
Item 3 is the one that hurts. now() inside a transformation makes your pipeline non-replayable, and nobody notices until a replay produces different numbers. The rule: a replayable pipeline is a pure function of its retained inputs. Anything that reads ambient, mutable or time-dependent context breaks the purity. Treat it as seriously as a correctness bug, because it is one, only deferred.
The requirement is determinism (the same input always gives the same output), because the index is a cache of the function's output. Nothing in the system tracks the function's definition as a cache key. Two failure modes follow directly.
From Indexing › Expression indexes
Where it appears
In 9 mechanisms. Each says what the mechanism is about, then how this principle shows up there, in its own words.
- Data Movement
How do records cross between two systems that have independent rates, independent availability and independent lifetimes, and what are you willing to lose when the assumption underneath the crossing breaks?
The rule: a replayable pipeline is a pure function of its retained inputs.
Read it inReplay
- Indexing
When does a separate "where is it?" structure pay for itself, and what does it cost to keep it true?
The requirement is determinism (the same input always gives the same output), because the index is a cache of the function's output.
Read it inExpression indexes: the column was always a degenerate case
- Emergent Failure Dynamics
What happens when nothing is broken, every component is healthy, and the system's own response to stress makes the stress worse?
Adding capacity during overload first removes capacity.
Read it inAutoscaling oscillationCold-cache collapseBoundedness plus hysteresisWhat metastability forbids
- Cost Mechanics
What is metered, and what is the smallest increment you are charged for?
Warm-up time is metered and produces no useful work.
Read it inScale-to-zero: a one-line break-even
- Time & Temporal Semantics
When am I allowed to act on data that may still be incomplete, and what do I owe the reader if I was wrong?
The logical time of a run is an input parameter, never an ambient read.
Read it inMonotonic clocks, wall clocks and replay determinism
- Streaming
What does it cost to remember, when the input never ends?
In a stream job, exactly-once is idempotence plus a consistent cut, and the checkpoint is the commit: its interval is your latency floor, and its duration sets your transaction timeout.
Read it inExactly-once: checkpoint plus transactional sinkStream-table joins and enrichmentSavepoints and state schema evolutionStateless operators: the part that is still just batch
- Schema Management & Evolution
What happens when the shape of your data changes on its own schedule, and nobody can change the readers, the writers and the old data at the same moment?
So the schema is non-deterministic across runs over identical data.
Read it inSchema inference and drift
- Orchestration & Workflow
How do you make unreliable, unobservable, irreversible side effects behave like a pure function of a time interval?
So the runtime must intercept all nondeterminism: time, randomness, I/O and concurrency.
Read it inDurable execution
- Measurement & Experimental Method
What would have to be true for this number to mean what I think it means?
An unpinned run is dated, not reproducible.
Read it inPinning, isolation and cold versus warm
Related
Made on the same pages most often.