Skip to content
About
Main content

Plan a sort

The decision

Some part of a query needs data in order: an ORDER BY, a merge , a , a sorted file layout. A sort produces a property that several later operators can share, but it must read all its input before producing anything. Is a sort needed here, and how should it be done?

You will end with whether a sort is needed, which order to buy, and its passes, fan-in and partitioning.


Work through it

1

Is order needed, or only grouping?

Grouping (all rows with the same key together, in any order) is strictly cheaper than order, and the two are routinely confused. If only grouping is needed: use hashing, not a sort.

2

Can the order be had for free?

A clustered layout, an index scan, an earlier merge join, an LSM-tree's read order, a . If the data already arrives in the needed order: use it. Check before buying.

3

Count the requirements one order satisfies

One sort order can serve several operators: an order by (a, b) also satisfies a requirement on a alone, and a column fixed to a constant can be ignored. If one candidate order satisfies two or more requirements: sort. If exactly one: compare it directly with the hash alternative.

4

Size the sort

With N bytes of data and M of memory: if N fits, sort in memory. Otherwise passes = 1 + ⌈log_F(N ÷ M)⌉, where the F = M ÷ buffer size B; a third pass is needed only when N > M²/B, and a normal two-pass sort does about 4N of I/O. With 16 GB of memory and 8 MB buffers, a third pass starts only past 32 TB. Choose the fan-in on purpose: a larger one means fewer passes but more comparisons per record.

5

If the sort is CPU-bound, normalize the keys

If rows are wide or keys are strings with language-aware collation: convert each key once into bytes that compare correctly with a plain byte comparison, and sort compact (key prefix, pointer) pairs instead of whole rows, unless gathering the rows afterwards would be random I/O.

6

If the sort is distributed, sample properly

Sample the data to choose the boundaries between machines, never by taking the first records of input that is already ordered. Size the sample for the imbalance you can accept. Check for one key appearing more than N ÷ machines times: it defeats exactly as it defeats hashing.

7

If a partial order exists, sort within segments

If data is already ordered by a prefix of the key (sorted by day, needing day then customer): sort each segment separately instead of the whole input.