Why the operating system's answer was rejected
Why does every serious data system turn swap off and take the memory decision back from the kernel?
The OS does offer a way to take memory back from a holder in the middle of its use. It is called swap. And swap is the first thing every serious data system turns off. Why? The answer gives this mechanism's second organizing idea.
Swap chooses which pages to throw out by looking at recency: pages not used lately are written to disk first. The kernel has no other signal. It sees a stream of page faults (a page fault happens when the program touches a page that is not in RAM). It cannot see the following three facts, which the application knows:
- These 8 KB are a hash-table bucket that will be probed once more in 40 ms.
- Those 8 KB are a sort run that has been fully merged and is now dead.
- That other 8 KB is an index root that every query today will touch.
The application knows all three. So the application is strictly better at the decision, and every mechanism here is a case of the application taking the decision back from the layer below:
| The layer below offers | The application replaces it with | Because the application knows |
|---|---|---|
| OS page cache | buffer pool | which pages are dirty, pinned, and about to be re-read |
| swap | spill-to-disk | which bytes are dead, and which can be written as a structured run rather than as arbitrary pages |
malloc/free | arenas, region allocation | that a whole phase's allocations die at the same instant |
| tracing garbage collector | off-heap / manual buffers | liveness by construction, without proving it by traversal |
| per-object allocation | Arrow-style pooled, reference-counted buffers | the shape and lifetime of a column in advance |
| kernel scheduling of residency | pin / unpin | what the code is in the middle of reading |
That table is this mechanism. Engineers call this idea the end-to-end argument: the layer closest to the real goal can make the best decision. Here it appears in a resource-management form. Note the price. Every row also takes a correctness burden back from the layer below:
mallocnever leaked memory on you; your arena can.- The garbage collector never gave you a use-after-free (using memory after it has been freed); your off-heap buffer will.
- The kernel never killed you for a wrong estimate of your memory need; your memory grant will.
The governing question of this mechanism: who is allowed to hold how much memory, for how long, and what happens to the work when the answer is less than it asked for.