Model

How wide a shuffle can be

Shuffle width grows as the square root of data size.

b = \frac{S}{MR}, M = R \leq \frac{S}{b _{m i n}}

$S$: Shuffle bytes; the total shuffle bytes
$M, R$: Width; producers (map tasks) and consumers (reduce tasks)
$b$: Bytes per connection; bytes each producer–consumer pair moves
$b_{m i n}$: Substrate floor; the minimum efficient transfer of the storage underneath

Recurring principle: Granularity mismatch

How it behaves

One input on the horizontal axis, the others held fixed.

NVMe floor ≈ 250 KBdisk floor ≈ 2 MBobject storage ≈ 4 MB

Data table

Widest square shuffle that stays above the floor, by shuffle size
Shuffle bytes S	series	M = R at most
954 MB	NVMe floor ≈ 250 KB	63.2
1.08 GB	NVMe floor ≈ 250 KB	68
1.24 GB	NVMe floor ≈ 250 KB	73
1.43 GB	NVMe floor ≈ 250 KB	78.5
1.66 GB	NVMe floor ≈ 250 KB	84.3
1.91 GB	NVMe floor ≈ 250 KB	90.6
2.21 GB	NVMe floor ≈ 250 KB	97.4
2.55 GB	NVMe floor ≈ 250 KB	105
2.95 GB	NVMe floor ≈ 250 KB	112
3.4 GB	NVMe floor ≈ 250 KB	121
3.93 GB	NVMe floor ≈ 250 KB	130
4.54 GB	NVMe floor ≈ 250 KB	140
5.24 GB	NVMe floor ≈ 250 KB	150
6.05 GB	NVMe floor ≈ 250 KB	161
6.98 GB	NVMe floor ≈ 250 KB	173
8.06 GB	NVMe floor ≈ 250 KB	186
9.31 GB	NVMe floor ≈ 250 KB	200
10.8 GB	NVMe floor ≈ 250 KB	215
12.4 GB	NVMe floor ≈ 250 KB	231
14.3 GB	NVMe floor ≈ 250 KB	248
16.6 GB	NVMe floor ≈ 250 KB	267
19.1 GB	NVMe floor ≈ 250 KB	287
22.1 GB	NVMe floor ≈ 250 KB	308
25.5 GB	NVMe floor ≈ 250 KB	331
29.5 GB	NVMe floor ≈ 250 KB	356
34 GB	NVMe floor ≈ 250 KB	382
39.3 GB	NVMe floor ≈ 250 KB	411
45.4 GB	NVMe floor ≈ 250 KB	441
52.4 GB	NVMe floor ≈ 250 KB	474
60.5 GB	NVMe floor ≈ 250 KB	510
69.8 GB	NVMe floor ≈ 250 KB	548
80.6 GB	NVMe floor ≈ 250 KB	589
93.1 GB	NVMe floor ≈ 250 KB	632
108 GB	NVMe floor ≈ 250 KB	680
124 GB	NVMe floor ≈ 250 KB	730
143 GB	NVMe floor ≈ 250 KB	785
166 GB	NVMe floor ≈ 250 KB	843
191 GB	NVMe floor ≈ 250 KB	906
221 GB	NVMe floor ≈ 250 KB	974
255 GB	NVMe floor ≈ 250 KB	1.05K
295 GB	NVMe floor ≈ 250 KB	1.12K
340 GB	NVMe floor ≈ 250 KB	1.21K
393 GB	NVMe floor ≈ 250 KB	1.3K
454 GB	NVMe floor ≈ 250 KB	1.4K
524 GB	NVMe floor ≈ 250 KB	1.5K
605 GB	NVMe floor ≈ 250 KB	1.61K
698 GB	NVMe floor ≈ 250 KB	1.73K
806 GB	NVMe floor ≈ 250 KB	1.86K
931 GB	NVMe floor ≈ 250 KB	2K
1.05 TB	NVMe floor ≈ 250 KB	2.15K
1.21 TB	NVMe floor ≈ 250 KB	2.31K
1.4 TB	NVMe floor ≈ 250 KB	2.48K
1.62 TB	NVMe floor ≈ 250 KB	2.67K
1.87 TB	NVMe floor ≈ 250 KB	2.87K
2.16 TB	NVMe floor ≈ 250 KB	3.08K
2.49 TB	NVMe floor ≈ 250 KB	3.31K
2.88 TB	NVMe floor ≈ 250 KB	3.56K
3.32 TB	NVMe floor ≈ 250 KB	3.82K
3.84 TB	NVMe floor ≈ 250 KB	4.11K
4.43 TB	NVMe floor ≈ 250 KB	4.41K
5.11 TB	NVMe floor ≈ 250 KB	4.74K
5.91 TB	NVMe floor ≈ 250 KB	5.1K
6.82 TB	NVMe floor ≈ 250 KB	5.48K
7.88 TB	NVMe floor ≈ 250 KB	5.89K
9.09 TB	NVMe floor ≈ 250 KB	6.32K
10.5 TB	NVMe floor ≈ 250 KB	6.8K
12.1 TB	NVMe floor ≈ 250 KB	7.3K
14 TB	NVMe floor ≈ 250 KB	7.85K
16.2 TB	NVMe floor ≈ 250 KB	8.43K
18.7 TB	NVMe floor ≈ 250 KB	9.06K
21.6 TB	NVMe floor ≈ 250 KB	9.74K
24.9 TB	NVMe floor ≈ 250 KB	10.5K
28.8 TB	NVMe floor ≈ 250 KB	11.2K
33.2 TB	NVMe floor ≈ 250 KB	12.1K
38.4 TB	NVMe floor ≈ 250 KB	13K
44.3 TB	NVMe floor ≈ 250 KB	14K
51.1 TB	NVMe floor ≈ 250 KB	15K
59.1 TB	NVMe floor ≈ 250 KB	16.1K
68.2 TB	NVMe floor ≈ 250 KB	17.3K
78.8 TB	NVMe floor ≈ 250 KB	18.6K
90.9 TB	NVMe floor ≈ 250 KB	20K
954 MB	disk floor ≈ 2 MB	22.4
1.08 GB	disk floor ≈ 2 MB	24
1.24 GB	disk floor ≈ 2 MB	25.8
1.43 GB	disk floor ≈ 2 MB	27.7
1.66 GB	disk floor ≈ 2 MB	29.8
1.91 GB	disk floor ≈ 2 MB	32
2.21 GB	disk floor ≈ 2 MB	34.4
2.55 GB	disk floor ≈ 2 MB	37
2.95 GB	disk floor ≈ 2 MB	39.8
3.4 GB	disk floor ≈ 2 MB	42.7
3.93 GB	disk floor ≈ 2 MB	45.9
4.54 GB	disk floor ≈ 2 MB	49.3
5.24 GB	disk floor ≈ 2 MB	53
6.05 GB	disk floor ≈ 2 MB	57
6.98 GB	disk floor ≈ 2 MB	61.2
8.06 GB	disk floor ≈ 2 MB	65.8
9.31 GB	disk floor ≈ 2 MB	70.7
10.8 GB	disk floor ≈ 2 MB	76
12.4 GB	disk floor ≈ 2 MB	81.7
14.3 GB	disk floor ≈ 2 MB	87.7
16.6 GB	disk floor ≈ 2 MB	94.3
19.1 GB	disk floor ≈ 2 MB	101
22.1 GB	disk floor ≈ 2 MB	109
25.5 GB	disk floor ≈ 2 MB	117
29.5 GB	disk floor ≈ 2 MB	126
34 GB	disk floor ≈ 2 MB	135
39.3 GB	disk floor ≈ 2 MB	145
45.4 GB	disk floor ≈ 2 MB	156
52.4 GB	disk floor ≈ 2 MB	168
60.5 GB	disk floor ≈ 2 MB	180
69.8 GB	disk floor ≈ 2 MB	194
80.6 GB	disk floor ≈ 2 MB	208
93.1 GB	disk floor ≈ 2 MB	224
108 GB	disk floor ≈ 2 MB	240
124 GB	disk floor ≈ 2 MB	258
143 GB	disk floor ≈ 2 MB	277
166 GB	disk floor ≈ 2 MB	298
191 GB	disk floor ≈ 2 MB	320
221 GB	disk floor ≈ 2 MB	344
255 GB	disk floor ≈ 2 MB	370
295 GB	disk floor ≈ 2 MB	398
340 GB	disk floor ≈ 2 MB	427
393 GB	disk floor ≈ 2 MB	459
454 GB	disk floor ≈ 2 MB	493
524 GB	disk floor ≈ 2 MB	530
605 GB	disk floor ≈ 2 MB	570
698 GB	disk floor ≈ 2 MB	612
806 GB	disk floor ≈ 2 MB	658
931 GB	disk floor ≈ 2 MB	707
1.05 TB	disk floor ≈ 2 MB	760
1.21 TB	disk floor ≈ 2 MB	817
1.4 TB	disk floor ≈ 2 MB	877
1.62 TB	disk floor ≈ 2 MB	943
1.87 TB	disk floor ≈ 2 MB	1.01K
2.16 TB	disk floor ≈ 2 MB	1.09K
2.49 TB	disk floor ≈ 2 MB	1.17K
2.88 TB	disk floor ≈ 2 MB	1.26K
3.32 TB	disk floor ≈ 2 MB	1.35K
3.84 TB	disk floor ≈ 2 MB	1.45K
4.43 TB	disk floor ≈ 2 MB	1.56K
5.11 TB	disk floor ≈ 2 MB	1.68K
5.91 TB	disk floor ≈ 2 MB	1.8K
6.82 TB	disk floor ≈ 2 MB	1.94K
7.88 TB	disk floor ≈ 2 MB	2.08K
9.09 TB	disk floor ≈ 2 MB	2.24K
10.5 TB	disk floor ≈ 2 MB	2.4K
12.1 TB	disk floor ≈ 2 MB	2.58K
14 TB	disk floor ≈ 2 MB	2.77K
16.2 TB	disk floor ≈ 2 MB	2.98K
18.7 TB	disk floor ≈ 2 MB	3.2K
21.6 TB	disk floor ≈ 2 MB	3.44K
24.9 TB	disk floor ≈ 2 MB	3.7K
28.8 TB	disk floor ≈ 2 MB	3.98K
33.2 TB	disk floor ≈ 2 MB	4.27K
38.4 TB	disk floor ≈ 2 MB	4.59K
44.3 TB	disk floor ≈ 2 MB	4.93K
51.1 TB	disk floor ≈ 2 MB	5.3K
59.1 TB	disk floor ≈ 2 MB	5.7K
68.2 TB	disk floor ≈ 2 MB	6.12K
78.8 TB	disk floor ≈ 2 MB	6.58K
90.9 TB	disk floor ≈ 2 MB	7.07K
954 MB	object storage ≈ 4 MB	15.8
1.08 GB	object storage ≈ 4 MB	17
1.24 GB	object storage ≈ 4 MB	18.3
1.43 GB	object storage ≈ 4 MB	19.6
1.66 GB	object storage ≈ 4 MB	21.1
1.91 GB	object storage ≈ 4 MB	22.7
2.21 GB	object storage ≈ 4 MB	24.3
2.55 GB	object storage ≈ 4 MB	26.2
2.95 GB	object storage ≈ 4 MB	28.1
3.4 GB	object storage ≈ 4 MB	30.2
3.93 GB	object storage ≈ 4 MB	32.5
4.54 GB	object storage ≈ 4 MB	34.9
5.24 GB	object storage ≈ 4 MB	37.5
6.05 GB	object storage ≈ 4 MB	40.3
6.98 GB	object storage ≈ 4 MB	43.3
8.06 GB	object storage ≈ 4 MB	46.5
9.31 GB	object storage ≈ 4 MB	50
10.8 GB	object storage ≈ 4 MB	53.7
12.4 GB	object storage ≈ 4 MB	57.7
14.3 GB	object storage ≈ 4 MB	62
16.6 GB	object storage ≈ 4 MB	66.7
19.1 GB	object storage ≈ 4 MB	71.7
22.1 GB	object storage ≈ 4 MB	77
25.5 GB	object storage ≈ 4 MB	82.7
29.5 GB	object storage ≈ 4 MB	88.9
34 GB	object storage ≈ 4 MB	95.5
39.3 GB	object storage ≈ 4 MB	103
45.4 GB	object storage ≈ 4 MB	110
52.4 GB	object storage ≈ 4 MB	119
60.5 GB	object storage ≈ 4 MB	127
69.8 GB	object storage ≈ 4 MB	137
80.6 GB	object storage ≈ 4 MB	147
93.1 GB	object storage ≈ 4 MB	158
108 GB	object storage ≈ 4 MB	170
124 GB	object storage ≈ 4 MB	183
143 GB	object storage ≈ 4 MB	196
166 GB	object storage ≈ 4 MB	211
191 GB	object storage ≈ 4 MB	227
221 GB	object storage ≈ 4 MB	243
255 GB	object storage ≈ 4 MB	262
295 GB	object storage ≈ 4 MB	281
340 GB	object storage ≈ 4 MB	302
393 GB	object storage ≈ 4 MB	325
454 GB	object storage ≈ 4 MB	349
524 GB	object storage ≈ 4 MB	375
605 GB	object storage ≈ 4 MB	403
698 GB	object storage ≈ 4 MB	433
806 GB	object storage ≈ 4 MB	465
931 GB	object storage ≈ 4 MB	500
1.05 TB	object storage ≈ 4 MB	537
1.21 TB	object storage ≈ 4 MB	577
1.4 TB	object storage ≈ 4 MB	620
1.62 TB	object storage ≈ 4 MB	667
1.87 TB	object storage ≈ 4 MB	717
2.16 TB	object storage ≈ 4 MB	770
2.49 TB	object storage ≈ 4 MB	827
2.88 TB	object storage ≈ 4 MB	889
3.32 TB	object storage ≈ 4 MB	955
3.84 TB	object storage ≈ 4 MB	1.03K
4.43 TB	object storage ≈ 4 MB	1.1K
5.11 TB	object storage ≈ 4 MB	1.19K
5.91 TB	object storage ≈ 4 MB	1.27K
6.82 TB	object storage ≈ 4 MB	1.37K
7.88 TB	object storage ≈ 4 MB	1.47K
9.09 TB	object storage ≈ 4 MB	1.58K
10.5 TB	object storage ≈ 4 MB	1.7K
12.1 TB	object storage ≈ 4 MB	1.83K
14 TB	object storage ≈ 4 MB	1.96K
16.2 TB	object storage ≈ 4 MB	2.11K
18.7 TB	object storage ≈ 4 MB	2.27K
21.6 TB	object storage ≈ 4 MB	2.43K
24.9 TB	object storage ≈ 4 MB	2.62K
28.8 TB	object storage ≈ 4 MB	2.81K
33.2 TB	object storage ≈ 4 MB	3.02K
38.4 TB	object storage ≈ 4 MB	3.25K
44.3 TB	object storage ≈ 4 MB	3.49K
51.1 TB	object storage ≈ 4 MB	3.75K
59.1 TB	object storage ≈ 4 MB	4.03K
68.2 TB	object storage ≈ 4 MB	4.33K
78.8 TB	object storage ≈ 4 MB	4.65K
90.9 TB	object storage ≈ 4 MB	5K

Worked examples, recomputed

Every number the mechanisms quote for this relationship is recomputed from it, so none of them disagrees with it.

Where it appears

Each mechanism states it in its own names. The general symbol each name stands for is shown with it.

Execution Engine Runtime
Once a query plan is chosen, why does the machine still sit idle, and what does each mechanism of a query runtime pay to buy that idle time back?
$b = S / (M \times R), M = R \leq S / b_{m i n}$
Read it in place →
Execution Engine Runtime
Once a query plan is chosen, why does the machine still sit idle, and what does each mechanism of a query runtime pay to buy that idle time back?
$bytes per connection = total shuffle bytes \div (M \times R)$
Read it in place →