Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Rack and zone awareness

Why is the latency hierarchy also the failure hierarchy, and when should a job concentrate rather than spread?

Here is the structural fact that makes placement hard. It is not usually stated plainly:

Same rack means same switch and same power. Same means same building. What is near is what dies together.

  • Latency wants concentration (close together is fast).
  • Availability wants dispersion (spread apart survives).
  • Cross-AZ transfer is billed, so cost votes with latency.

Two of the three forces push toward concentration. That is why zone spreading is always a policy imposed from outside, and never an emergent behavior of an optimizing scheduler.

The non-obvious corollary: OR-jobs versus AND-jobs

Whether spreading helps depends on whether your job is an OR or an AND over its parts.

  • A replicated service survives if any replica lives. That is an OR. Spreading raises availability.
  • A gang job completes only if all tasks live. That is an AND. Spreading over k domains increases the chance that at least one domain event touches you, so spreading lowers the completion probability.

Same topology, opposite optimum, decided entirely by the job's internal coupling. Tightly coupled training jobs are placed deliberately tight. Stateless serving is placed deliberately wide. Neither is a preference. (Copies of data follow durability's arithmetic instead: placement across failure domains.)

Proximity and shared fate are one tree read in opposite directions.