Fail-fast vs. fail-degraded
Reject or continue when something is wrong.
Where you meet it: strict parsing vs. permissive, stop-the-line vs. quarantine, load shedding vs. queueing, circuit break vs. retry.
What it is
Two more points shape every decision here. First, every check is compute inserted into the pipeline's hot path, so check depth trades directly against pipeline latency. Second, every mechanism here is a position on one choice: fail fast (reject the data and stop) or fail degraded (continue, with a warning or with part of the data). Also, the time of binding decides where a defect shows up: if you check at write time, it surfaces in the producer; if you never check until read time, it surfaces in the tenth downstream consumer.
From Data Quality & Validation › Trust boundaries
The mistake almost everyone makes early is to treat fail-degraded as the kinder option. But a degraded system that never recovers is worse than one that rejected 10% of the load and stayed healthy.
From Data Movement › Four answers to overload
What does this system assume that it cannot verify? That the distribution of the data still looks like the data used to choose the level, the block size and the dictionary. Nothing checks this. When a new customer arrives with UUIDs where there used to be a few enum values, the ratio silently halves, and the result shows up months later as storage growth and slower queries, never as an error. Compression fails by degrading, and that is the kind of failure that organizations are worst at detecting. You should alert on compression ratio per column as a data-quality signal. Almost nobody does.
From Encoding, Serialization & Compression › What compression forbids
Where it appears
In 13 mechanisms. Each says what the mechanism is about, then how this trade-off shows up there, in its own words.
- Physical Substrate & Performance Laws
Why do trade-offs exist at all?
Load shedding is how you refuse to operate beyond the knee.
Read it inLittle's Law and the queueing wallTail at scale, and why percentiles do not combine
- Data Movement
How do records cross between two systems that have independent rates, independent availability and independent lifetimes, and what are you willing to lose when the assumption underneath the crossing breaks?
This whole article is one trade-off seen from different sides: fail-fast versus fail-degraded.
Read it inArrival exceeds service: four answersWhat changed: from one machine to twoDead-letter queues
- Encoding, Serialization & Compression
What do we turn our values into when we turn them into bytes, and what does that choice cost?
The blast radius of an error is the block size.
Read it inBlock size, level and trained dictionariesWhat general-purpose compression forbidsSerialization: where the description lives
- Query Processing
Why can one question be answered by plans whose costs differ by a factor of ten thousand, and what does each operator force the engine to do?
That is a fail-fast escape, and it is the one exception to the earlier statement that approximation is the only degradation mode: this is the single place where the engine simply stops with an error.
Read it inSet operators, pivots and recursionSet and dedup operations: aggregation with an empty accumulatorFrom what to how: why a query has many plans
- Memory & Resource Management
Who is allowed to hold how much memory, for how long, and what happens to the work when the answer is less than it asked for?
Spilling buys survival by converting a memory problem into an I/O problem and a pipelining problem.
Read it inSpill is algorithm substitutionMemory grants and reservations
- Emergent Failure Dynamics
What happens when nothing is broken, every component is healthy, and the system's own response to stress makes the stress worse?
The general principle: a queue must have a discard policy.
Read it inQueue amplification and unbounded bufferingRetry amplificationFail-stop as a purchased assumptionA decision that has no answerWhy restarting sometimes worksGraceful degradation and brownoutYou cannot defend by doing more work
- Scheduling & Workload Management
Which work runs now, which work waits, and who pays for the waiting?
Admission control is fail-fast in its cleanest form, and the earlier a failure happens, the cheaper it is.
Read it inAdmission control and queuing
- Time & Temporal Semantics
When am I allowed to act on data that may still be incomplete, and what do I owe the reader if I was wrong?
The idleness timeout is a deliberate trade of correctness for liveness (keeping the pipeline moving).
Read it inWatermarks: declaring completeness
- Change Data Capture & Incremental Processing
What does it cost to know what changed, and is that cheaper than not needing to know?
Always overlap deliberately, and make the sink idempotent so the overlap is free.
Read it inBackfill and cutover
- Schema Management & Evolution
What happens when the shape of your data changes on its own schedule, and nobody can change the readers, the writers and the old data at the same moment?
There are three places to detect a breaking change, with roughly an order of magnitude in cost between each:
- Data Quality & Validation
How do you check that data is right, when "right" is a claim about the world and not about the bytes?
The safe stop duration is bounded by your provisioned over-capacity.
Read it inStop-the-line, and why the factory analogy brings in the wrong cost model
- Orchestration & Workflow
How do you make unreliable, unobservable, irreversible side effects behave like a pure function of a time interval?
An approval that never comes is a workflow stuck forever.
Read it inHuman-in-the-loop stepsDAGs, dependency resolution and trigger rules
- Observability & Operations
What is the smallest description of a running system from which we can find a fault, fast enough to matter, at a cost we can still pay while the system is failing?
When the collector is saturated, the correct behaviour is to drop telemetry, never to apply backpressure into the application (slow the application down until the collector catches up).
Read it inThe telemetry pipeline is a data system
Related
Made on the same pages most often.