Stop-the-line, and why the factory analogy brings in the wrong cost model
What does halting a pipeline really cost, and why does the factory analogy mislead?
A circuit breaker is the strongest gate: halt propagation when quality fails. The justification is the fan-out arithmetic: halting at the boundary caps both the detection-latency term and the consumer-count term in the defect cost formula.
The cost side is where the interesting derivation is. A data pipeline cannot stop the arrival of work, only its processing. Stop for six hours and you now have a backlog.
By the catch-up condition, a stop of T hours at utilization u takes T × u ÷ (1 − u) hours to drain. A six-hour stop at 60% utilization costs 9 hours of catch-up; the same stop at 90% costs 54 hours. The same six-hour stop is a nuisance at 60% and a multi-day incident at 90%. That arithmetic is linear in T; the cost becomes superlinear through thresholds, when the recovery load is larger than what the system can absorb.
If steady-state utilization is high, the restart is itself an incident: a backlog burst, a replay storm, and metastable behaviour, where the recovery load re-triggers the same failure that caused the stop and the system cannot return to normal by itself. The safe stop duration is bounded by your provisioned over-capacity. The true cost of a circuit breaker is therefore paid in advance, as permanent over-provisioning.
Why the assembly-line analogy fails
This corrects the andon cord analogy (in a car factory any worker can pull a cord to stop the line when they see a defect). It is repeated far too confidently. On an assembly line, stopping is cheap, because input stops with the line and work-in-progress is physically bounded. Neither holds for data: your input is exogenous (it comes from outside and does not stop), your work-in-progress is unbounded, and the "line" fans out to consumers who took no part in the decision. The intuition transfers. The cost model does not.
Scope reduction: break the circuit by partition
The mechanism that makes stopping survivable is scope reduction. Break the circuit at partition granularity (one region, one day) and not at dataset granularity, so that one bad region or one bad day stops while the rest flows. This is bulkheading (separating compartments so a failure in one does not sink the others) applied to correctness instead of load. It is the same trade: you lose the simplicity of a single verdict, and you gain a blast radius smaller than the whole system.
Compression: stopping caps defect propagation, but every hour stopped costs u ÷ (1 − u) hours of catch-up, and more once recovery load triggers new failures, so a circuit breaker is only as safe as your spare capacity.