Delay scheduling
When is waiting for a local slot worth it, and where did the mechanism move once disks stopped mattering?
The original reasoning is sound, but the mechanism is now mostly "dead engineering." It is worth separating the two carefully.
The original situation (the Hadoop/HDFS era). A task reading data from its local disk ran at disk bandwidth. A task reading remotely had to cross a rack uplink that was oversubscribed (more server bandwidth under the rack than uplink capacity out of it) by 5–20 to 1. So the effective locality ratio was around one order of magnitude: local was about 10× faster.
The trick. Delay scheduling exploits an asymmetry between two waits. If the cluster runs N short tasks of mean length T, then some slot frees roughly every T/N seconds. So the expected wait to get one of your k local slots is small, while the gain is the locality ratio applied to the whole task. So: skip your turn briefly, then take the local slot. The rule is:
- how much slower the task runs away from its data
- slots that hold the task's data
- mean task length
The condition is easily satisfied when tasks are short and the ratio is 10.
Both inputs then collapsed. Separating compute from storage meant every read is remote for everyone. And 25–100 Gb network cards, plus the latency floor of object storage, flattened the ratio to near 1. There is no data-locality gradient left to schedule against.
But the mechanism did not die. It moved. The gap it exploits now sits one tier up, between cached and uncached: executor-local cached blocks, the operating-system page cache, shuffle files on local NVMe, warm containers. Memory versus network is again 10–100 to 1. This gives a sharper statement:
Locality scheduling does not track disks. It tracks whichever tier currently has the largest gap to the tier below it.
When one layer gets faster, the mechanism relocates rather than disappearing. Delay scheduling was never about disks. It is about waiting a short time for a large ratio, and the ratio has migrated upward.