Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

The machine becomes a location

In which four ways do machines differ, and why do those ways disagree?

Deciding when asked when, over a pool treated as uniform. Placement drops that assumption. Machines differ in four ways at once:

  1. what data is near them (locality),
  2. what fails together with them (failure domains),
  3. what they cost,
  4. what shape they are (how much CPU, memory, GPU).

And the four disagree with each other. So the decision changes from an ordering over one dimension to a matching against a conjunction of constraints. Conjunctions are multiplicative: every constraint you add shrinks the feasible set, and the scheduler's job quietly turns from optimization into search for anything legal at all.

Two more facts shape every placement below. Placement is online: it is bin packing without knowledge of future arrivals, the same missing information as unknown job durations, now in space instead of time. And a stateful task (one that holds data in memory or on local disk) has its placement re-decided at every failure, each move carrying a state-transfer cost that stateless work, which restarts anywhere, never pays (moving stored data is coordination's).