Model
Scan time with per-file overhead
The first term is overhead: it grows with the number of files, not with the data.
- Files
- files the scan opens
- Per-file cost
- fixed per-object cost (round trip, footer, metadata), 20–50 ms on an object store
- Parallelism
- how many files are fetched at once
- Data
- bytes of data read
- Bandwidth
- aggregate read bandwidth
How it behaves
One input on the horizontal axis, the others held fixed.
Data table
| File size | Scan time |
|---|---|
| 97.7 KB | 166 s |
| 110 KB | 149 s |
| 123 KB | 134 s |
| 138 KB | 121 s |
| 155 KB | 109 s |
| 174 KB | 97.9 s |
| 195 KB | 88.3 s |
| 219 KB | 79.8 s |
| 245 KB | 72.2 s |
| 275 KB | 65.4 s |
| 309 KB | 59.4 s |
| 346 KB | 54 s |
| 389 KB | 49.2 s |
| 436 KB | 45 s |
| 489 KB | 41.2 s |
| 549 KB | 37.8 s |
| 616 KB | 34.8 s |
| 691 KB | 32.1 s |
| 776 KB | 29.7 s |
| 870 KB | 27.5 s |
| 977 KB | 25.6 s |
| 1.07 MB | 23.9 s |
| 1.2 MB | 22.4 s |
| 1.35 MB | 21.1 s |
| 1.51 MB | 19.9 s |
| 1.7 MB | 18.8 s |
| 1.9 MB | 17.8 s |
| 2.14 MB | 17 s |
| 2.4 MB | 16.2 s |
| 2.69 MB | 15.5 s |
| 3.02 MB | 14.9 s |
| 3.38 MB | 14.4 s |
| 3.8 MB | 13.9 s |
| 4.26 MB | 13.5 s |
| 4.78 MB | 13.1 s |
| 5.36 MB | 12.8 s |
| 6.02 MB | 12.5 s |
| 6.75 MB | 12.2 s |
| 7.58 MB | 12 s |
| 8.5 MB | 11.8 s |
| 9.54 MB | 11.6 s |
| 10.7 MB | 11.4 s |
| 12 MB | 11.2 s |
| 13.5 MB | 11.1 s |
| 15.1 MB | 11 s |
| 17 MB | 10.9 s |
| 19 MB | 10.8 s |
| 21.4 MB | 10.7 s |
| 24 MB | 10.6 s |
| 26.9 MB | 10.6 s |
| 30.2 MB | 10.5 s |
| 33.8 MB | 10.4 s |
| 38 MB | 10.4 s |
| 42.6 MB | 10.3 s |
| 47.8 MB | 10.3 s |
| 53.6 MB | 10.3 s |
| 60.2 MB | 10.2 s |
| 67.5 MB | 10.2 s |
| 75.8 MB | 10.2 s |
| 85 MB | 10.2 s |
| 95.4 MB | 10.2 s |
| 107 MB | 10.1 s |
| 120 MB | 10.1 s |
| 135 MB | 10.1 s |
| 151 MB | 10.1 s |
| 170 MB | 10.1 s |
| 190 MB | 10.1 s |
| 214 MB | 10.1 s |
| 240 MB | 10.1 s |
| 269 MB | 10.1 s |
| 302 MB | 10 s |
| 338 MB | 10 s |
| 380 MB | 10 s |
| 426 MB | 10 s |
| 478 MB | 10 s |
| 536 MB | 10 s |
| 602 MB | 10 s |
| 675 MB | 10 s |
| 758 MB | 10 s |
| 850 MB | 10 s |
| 954 MB | 10 s |
Where it appears
Each mechanism states it in its own names. The general symbol each name stands for is shown with it.
- Table & Storage Maintenance
Who pays for the gap between the order data arrives in and the order it is read in: the writer, every reader, or a background job, and what is the break-even?