Where spill goes, and what that assumes

What breaks when the spill disk is remote, the node is elastic, or the data is regulated?

Every design in this article assumes fast local disk. There are four ways in which the surrounding assumptions break.

Disaggregation. On Kubernetes or in a cloud-native setup, "local disk" may be network-attached. That moves the bandwidth ratio r from about 50 to about 500. The formula of What spilling actually costs then gives, for 5% spilled: 0.95 + 2 × 0.05 × 500 = 0.95 + 50 = ~51×, instead of ~6×. The cliff is an order of magnitude steeper. The spill design is unchanged; its cost model is not. This is a case where a deployment decision silently invalidates an engine's tuning defaults.

Node affinity fights elasticity. Spill files live where the operator ran, so a node that holds spill state cannot be removed. Spill is state, and state pins capacity. An elastic worker becomes a stateful one while it spills. This puts spill in direct conflict with autoscaling. (Shuffle output has the same problem, and dedicated shuffle services exist to solve it.)

Compression is free and should be automatic. LZ4-class compression codecs run at several GB/s per core, which is comfortably faster than the device they feed. The CPU doing the compression would otherwise be stalled waiting for I/O. When r is large, compressing spill divides the dominant term by the compression ratio at almost no cost. For example, with a 3:1 ratio, the 2·f·r term in the formula divides by 3. This is one of the rare decisions with no real trade-off, because the break-even is passed by an order of magnitude. The only reason to think about it is when the data is already compressed or encrypted upstream and will not compress again.

Encryption is a compliance question that lands here. Spill files contain user data, written to local disk. They are the most commonly overlooked place where production data comes to rest unencrypted. In a regulated environment, encrypting them is a hard requirement. It adds a second CPU pass on the same path, and unlike compression this is a real cost, because encryption does not reduce the number of bytes it protects.

One-line compression: spill assumes fast local disk and produces node-affine state, two assumptions that cloud deployment breaks in opposite directions.

Case, resolvedConstructed case · Amara at Voltara

5% spilled, 6 times slower

A join that spilled only 5% of its build side ran six times slower, a join on a meter id shared by 300 million faulty readings never finished, every query spilled at once onto one NVMe drive, and an OOM-killed executor tripled the batch time. Where it started: Degradation →

5% spilled, 6 times slower. She computes the bandwidth ratio for the actual hardware: memory about 100 GB/s, NVMe about 2 GB/s, so r = 50. At f = 0.05 the formula gives about 6×. No bug. The fix is a larger grant for this join (quantized to the next pass boundary, Memory grants and reservations per operator) and LZ4 compression of spill files, which cuts the 5.0 term roughly by the compression ratio.
The hot key that never finishes. The faulty meters all report the id 0000000. Re-partitioning cannot split one key. She adds a sort-merge fallback for oversized partitions, and she rewrites the per-meter totals as partial (algebraic) aggregation: SUM and COUNT need no spill at all for the hot key.
Everything spills at once. Amara adds a limit on concurrent spilling queries (a gate on spill bandwidth), so that the NVMe stays below about 70% utilization. Queries that cannot spill yet wait in the queue.
The OOM-killed executor. She changes the order of defences: the engine now kills the offending query at the hard watermark, long before the kernel would kill the process. The UDF-holding queries, her list of unspillable operators, are the first candidates. She also connects the kill path to grant feedback, so the killed query is retried with a larger grant and not with the same one.