Playbooks

Choose an execution model

A workload must run: on one machine, or across many as a pipelined or materialized dataflow, bulk-synchronous rounds, actors or serverless functions? Three questions, then the arithmetic that decides how failures are survived.

Plan the parallelism

Before adding workers: find the ceiling set by the largest partition, the stages set by pipeline breakers, whether the costly operator can be split at all, how to size tasks, and when speculation and elastic scaling pay.

Handle skew and elasticity

A stage's last tasks run far longer than the rest: plot duration against input bytes to tell imbalance, a straggler and skew apart, then apply the fix that works for that one.

Design a shuffle

Data must be repartitioned across machines: compute bytes per connection against the storage floor, cap the width, shrink the data first, then choose hash or sort, a shuffle service, compression and serialization.