Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Fail-slow hardware

Why is fail-slow the normal hardware failure mode, and how does masking cause it?

The evidence worth carrying is a field study by Gunawi and colleagues across large operators. It found:

  • disks delivering a small fraction of their rated for months,
  • network cards degraded by firmware bugs,
  • CPUs throttled by heat because of a failing fan (Wayhop's search node),
  • SSDs whose internal stalls writes for seconds,
  • memory degraded by correctable-error handling.

The prevalence is high enough that should be treated as the normal hardware failure mode, and as the fortunate one.

Masking converts failures into slowness

The cause is design, not bad luck. Every layer of the stack has been given internal redundancy and error correction to improve reliability, and each of these mechanisms converts a hard failure into a slow one:

  • NAND read-retry and ECC (error-correcting codes) turn an unreadable flash page into a slow page.
  • Sector remapping turns a bad disk sector into an extra seek.
  • RAID and erasure-coded rebuild turn a dead drive into a degraded array.
  • TCP retransmission turns packet loss into .
  • Connection pools turn a dead connection into a wait.
  • Retries turn a failed request into a slow one.

So the industry systematically moved its failures from the error channel to the latency channel. That was the right choice for , and it has a consequence nobody priced: the error channel has a schema, an observer and an alert; the latency channel has a distribution nobody is watching of. Errors are counted, logged, typed and paged on. Latency is averaged. A failure moved from the first channel to the second becomes invisible by that move alone.

So for any mechanism, ask what signal was consumed, and instrument that: counts, correction counts, rebuild state, fallback invocations, coercion counts. The masked path is precisely where a failure can persist indefinitely without an observer. The strong form: a masking mechanism without a counter is a mechanism for hiding failures, not for surviving them.