Absence is not evidence
Absence of an event is not absence of a failure.
What it is
When data moves between systems, the monitoring requirement is different in kind. In storage you monitor for errors. In movement you monitor for absence and asymmetry: counts that do not reconcile from end to end, a maximum that diverges from the mean, a lag that has stopped changing, an age that only grows. If your monitoring can only tell you that something went wrong, it is the wrong monitoring for this domain. It must tell you that something did not happen.
From Data Movement › The whole domain
You cannot derive this claim from the data you observe, because not having seen a record is not evidence that none exists (absence of an event is not absence of a failure). So the claim must be asserted by the system. This assertion is a watermark:
From Time & Temporal Semantics › Watermarks
You measure precision but never recall. Fired checks are visible and can be judged. Defects for which no check was written produce no evidence of their own absence. Every quality programme overestimates its coverage, and the bias is structural, not a lack of rigour. The only instrument that partly escapes it is a differential against an independent authority, which is why reconciliation exists as a separate discipline instead of being "more validation".
From Data Quality & Validation › Propositions about the world
Where it appears
In 16 mechanisms. Each says what the mechanism is about, then how this principle shows up there, in its own words.
- Data Movement
How do records cross between two systems that have independent rates, independent availability and independent lifetimes, and what are you willing to lose when the assumption underneath the crossing breaks?
It must tell you that something did not happen.
Read it inData movement, stepped back
- Consistency & Isolation Models
When many users, many objects and many copies exist at once, which histories does the system promise never to show, and who writes the code for the histories it does show?
Nothing logs it, nothing detects it, nothing can.
Read it inCausal consistency
- Distributed Systems Coordination
Who is allowed to decide, and how does everyone else learn the decision without asking?
Compression: a failure detector trades completeness against accuracy with a timeout, and its verdict describes a link, not a node.
Read it inFailure detection: liveness is a relation, not an attributeReplace unverifiable facts with verifiable onesResharding: the map is harder than the bytesA dead node looks like a slow oneOne missing present
- Emergent Failure Dynamics
What happens when nothing is broken, every component is healthy, and the system's own response to stress makes the stress worse?
The gap between throughput and goodput is the whole phenomenon.
Read it inCongestion collapseThundering herdGray failure in the data planeBimodality: why there is no middle
- Cost Mechanics
What is metered, and what is the smallest increment you are charged for?
Second: cost is the only failure mode with no failure detector.
- Time & Temporal Semantics
When am I allowed to act on data that may still be incomplete, and what do I owe the reader if I was wrong?
> It takes an ambiguous silence and resolves it, by decree, to the harmless explanation: "no data means nothing happened."
Read it inWatermarks: declaring completeness
- Streaming
What does it cost to remember, when the input never ends?
A stream-stream join's state is rate × window × both sides, and its errors are absences.
Read it inStream-stream joins
- Change Data Capture & Incremental Processing
What does it cost to know what changed, and is that cheaper than not needing to know?
Principle: observability of change is a property of the source, not of the observer.
Read it inPolling and the completeness gapWhat change data capture forbids, and what escaping costs
- Data Quality & Validation
How do you check that data is right, when "right" is a claim about the world and not about the bytes?
It is the only check that fires on the absence of data.
Read it inStatistical checks: from predicates to estimatorsPropositions about the worldTrust boundaries: where checks belongSampling: cheapest exactly where it is weakestWhat data quality can claim
- Metadata, Catalog & Lineage
What can a system prove about its data without reading the data, and how does it know that proof is still true?
The decay rate ranks strictly: construction ≪ observation ≪ assertion.
Read it inLoad-bearing versus advisoryProvenance annotationsFidelity and the absence asymmetryOwnership and stewardshipUsage and query-log mining
- Orchestration & Workflow
How do you make unreliable, unobservable, irreversible side effects behave like a pure function of a time interval?
The consequence is usually missed: SLA monitoring inside the orchestrator is structurally unsound for the case that matters most.
Read it inSLAs and absence detectionOrchestration moves nothingIdempotent task design and completion verificationEvent-driven triggering
- Observability & Operations
What is the smallest description of a running system from which we can find a fault, fast enough to matter, at a cost we can still pay while the system is failing?
Freshness is a property of the data, never of the job.
Read it inFreshnessCorrect per arrival, and progressDistributed tracingAlerting is a base-rate problem, not a detection problem
- Security & Privacy
How do you protect data from someone who watches your design and then chooses how to attack it?
The governing constraint is that identity is never self-certifying.
Read it inIdentity is never self-certifyingHashing and salting identifiersDetection and accountability
- Specialized Processing Models
What do you build when a query predicate cannot be reduced to a range over an ordered key?
One line: a gap does not record why it is a gap, so every fill is an unverifiable assumption.
- Interfaces & Formats
What can two parties agree on when they cannot talk to each other?
A protocol is a conversation with amnesia at both ends: everything it guarantees comes from what it forces you to say again.
Read it inA conversation has state; SQL as the interfacePagination, cursors and incremental sync tokens
- Measurement & Experimental Method
What would have to be true for this number to mean what I think it means?
Principle: every instrument has a blind region, and the cost tends to hide in it.
Read it inDifferential profilingHistory checking: testing a consistency promiseCoordinated omission and steady stateNoise floor, effect size, outliers and run order
Related
Made on the same pages most often.