Choose an aggregation strategy
A query groups rows and aggregates them: revenue per customer, median latency per endpoint, distinct users per day. The output can only be as large as the number of groups, so data can often be reduced before it moves across the network. Whether that is possible depends on the function and on how many rows share each group. Which plan fits?
You will end with a single- or two-phase plan (or an exact plan for a function that cannot be split), and how its state is sized.
Decide
Can the function be computed from partial results?
SUM, COUNT, MIN, MAX: yes. AVG: yes, as sum and count. Exact MEDIAN, exact percentiles, exact COUNT DISTINCT: no.
Why
Functions that combine from partials (distributive or algebraic) allow pre-aggregation, spilling partial results, and immunity to skewed groups. Holistic functions allow none of these; they need every value of the group in one place.
Is an exact answer required?
Why
A sketch (HyperLogLog for distinct counts, t-digest for percentiles) turns a holistic function into one that combines from partials, and with it every option above.
How many rows does each worker hold per group?
Rows ÷ (workers × groups). 1 billion rows across 100 workers into 10,000 groups is 1,000 per group per worker.
Above about 10Two-phase aggregation
Near 1Single-phase aggregation
Why
Pre-aggregating on each worker shrinks the data by that ratio before the shuffle. Above about 10, it dominates every other decision in the query. Near 1, every row is its own group, and the extra phase is pure overhead.
Where you can land
Two-phase aggregation
Use it when the function combines from partials and many rows share each group per worker.
Each worker aggregates its own rows, then partial results are shuffled by group and combined.
Single-phase aggregation
Use it when almost every row is its own group.
Shuffle rows by group and aggregate once.
Before you build it: check the engine decides this from a sample at run time, rather than trusting a plan-time estimate of the number of groups.
Exact plan for an unsplittable function
Use it when the function cannot combine from partials and the answer must be exact.
Either sort by (group, value), which makes exact medians and percentiles a matter of position with constant memory, or shuffle by the distinct column first, then by group: two shuffles, but parallel.
Whichever you land on
Size the state
State ≈ number of groups × size of each accumulator, against memory. If it spills: spill partial results if they can be merged, raw rows otherwise. Skewed groups reduce the spill, since a hot group is one entry.
Look for a free sort
If the query orders by the grouping key, or the input arrives already grouped: sort-based aggregation becomes the cheap option, not just the safe one.
Check determinism before someone reconciles the number
Floating-point sums depend on the order of addition, so parallel runs can differ in the last digits; some sketch merges are lossy. If the number will be reconciled against another system: use exact decimal types or a deterministic summation order.