The Data Engineering

Foundations

About Execution at Scale Backpressure Broadcast Variables Data Skew Join Strategies Lazy vs Eager Evaluation Memory Management Operator Chaining Query Planning Resource Contention Shuffles Aggregation Strategies Speculative Execution Spill to Disk Task Scheduling Vectorization

The Data Engineering

Foundations Execution at ScaleShuffles

Shuffles Are the Most Expensive Operation