Unspillable state and the kill decision
Whom do you kill when nothing can degrade?
Not everything can degrade. The list is short and worth knowing by heart, because these are the operators that turn a memory shortage into a failure instead of a slowdown:
- A window function over an unbounded frame (for example "running total since the beginning"): the whole partition must be in memory.
- A single aggregation group larger than memory with a holistic aggregate (the terminal case of Hash spill, and the arithmetic of a single pass).
- A UDF or UDAF (a user-defined aggregate function) that holds state the engine cannot write to disk. The engine usually cannot serialize it.
- The build side of a broadcast join (a join where a small table is sent in full to every worker). It is resident by definition. A broadcast threshold such as
spark.sql.autoBroadcastJoinThreshold, and its equivalents in other engines, is literally an admission gate on unspillable state, placed in the query planner instead of the memory manager.
When no degradation exists, the choice collapses to a choice of victim. The three options have different cost distributions:
| Choice | Memory recovered | Work destroyed | Fairness |
|---|---|---|---|
| Kill the requester | least | least | punishes the innocent arrival |
| Kill the largest holder | most | most | systematically punishes the important query |
| Let the OS kill the process | everything on the node | everything on the node, plus dependents | arbitrary |
The third row is much worse than it looks, and the reason is granularity mismatch one last time. The granularity of a failure determines its blast radius, and the OS's granularity is the process. When the OOM killer kills a Spark executor, you lose not only its running tasks. You also lose its cached blocks and its shuffle output (the intermediate files that other stages will read). That forces recomputation in upstream stages that had already finished. The cost is superlinear in what was lost. This asymmetry is why every engine works so hard to fail a query before the OS fails a process. The engine is buying a finer failure granularity, and paying for it with all the accounting machinery of Rationing. (This is Amara's fourth incident.)
The retry trap
A killed query that is retried unchanged will fail again in exactly the same way. It is a poison pill: a piece of work that keeps crashing whatever runs it. The only useful retry has a larger grant or a different plan. That requires the feedback mechanism from Where the grant number comes from: the most important item in this article.
Kill-and-retry without learning is an infinite loop that uses up a concurrency slot every time round. It produces goodput without throughput: lots of activity, no finished work. (The same distinction applies to aborted database transactions that are retried.)
One-line compression: when nothing can spill, the only question is which failure granularity you pay in, and the OS's granularity costs the most because it destroys work that had already succeeded.