Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Blast radius, failure domains and cells

Do cells reduce downtime, and if not, why are they still the right design?

The is the share of users or functions affected by one failure. A failure domain is a set of things that can fail together. The standard way to shrink the blast radius is to partition a global system into k independent instances called cells, each serving 1/k of the users, with no shared . A failure then affects one cell.

The honest arithmetic

The usual justification is wrong, so do the arithmetic. If the per-cell failure probability is unchanged, k cells fail k times as often, each failure affecting 1/k of users. The .

Why cells are still correct

Three reasons survive the arithmetic.

  1. Impact is strongly nonlinear in scope. A 5% outage is an incident. A 100% outage is an existential event. Nobody is indifferent between one of the latter and twenty of the former.
  2. Cells break the correlation on the dominant cause (this is the real mechanism). Deploys and configuration changes cause a large share of outages. Cells let you apply a change one cell at a time, converting a correlated global failure into a detectable local one, with an abort before the next cell. The reduction in failure probability comes from progressive deployment, which cells enable. It does not come from partitioning by itself.
  3. A cell is a bounded unit for capacity testing. You can actually know its limits, which you cannot for a system of unbounded size.

The costs

The costs are real: a k-fold loss of statistical , cross-cell operations that become hard or impossible, and a routing layer that needs a cell-assignment mechanism. That mechanism is a new shared dependency, and the one place where a global failure can still begin. Keep that router dumb, static and rarely changed, or you have moved the single point of failure instead of removing it.