Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Off-heap memory management

What does moving data off-heap really remove, and when is it a net loss?

The constraint

A tracing collector's mark cost is proportional to the live set, not to the garbage. Garbage is free to collect; survivors are what you pay for. Generational collection exploits the fact that most objects die young. But a cache is defined as objects that do not die young. So a large in-heap cache is the exact worst case for the design the collector is optimized around.

An order of magnitude. Tracing runs at about 10⁸ to 10⁹ references per second per thread. A 40 GB held as hundreds of millions of small objects means seconds of marking work for each full traversal.

Concurrent collectors (G1's concurrent mark; ZGC and Shenandoah with concurrent relocation) move that work out of the pause, but not off the machine. They pay in CPU, typically 10–20% of throughput, plus a read or write barrier (a small piece of code run on every reference operation of the program). Costs are moved, and the collectors' own design documents say so.

The mechanism

Represent bulk data as a small number of large, opaque byte regions that the collector cannot look into. One 1 GB direct buffer is one object to trace, instead of thirty million. Addressing inside it is offset arithmetic.

Three consequences are forced. The second is usually the biggest win.

  1. You now own the layout. Records must be serialized into a fixed-offset binary form. Spark's UnsafeRow is the standard example. Encoding decisions that a compression chapter would discuss now arrive inside the memory manager, because no language runtime decides object representation for you.
  2. Operators can work on bytes without creating objects. A sort that stores (8-byte key prefix, pointer) pairs and compares prefixes in place never follows the pointer on the common path. That is a CPU-cache locality win bought with a memory-manager mechanism, and in practice it often matters more than the saving that motivated the change.
  3. You own lifetime. Direct buffers freed through Cleaner or phantom references are freed when the collector gets around to noticing, which brings the collector back at the least convenient moment. So real systems use reference counts and free explicitly, and the leak is now yours.

"Off-heap avoids GC pauses" is imprecise in a way that causes bad decisions. The accurate claim is:

Off-heap removes objects from the traced set. That reduces mark cost and fragmentation, and it moves liveness from proved by traversal to known by construction.

For small short-lived objects it is a net loss. Allocation in the young generation is a pointer bump, and a copying collector touches only survivors, so garbage that dies young costs near zero. A copy across the heap boundary costs more than the collection it replaced.

Boundary condition

Off-heap wins when the data is (i) large, (ii) long-lived or phase-lived, and (iii) uniform in shape so that offset arithmetic is cheap. If any one of the three is missing, you have bought complexity and a new class of bug.

The garbage collector as an indirection layer

A managed runtime is an indirection layer (object references) plus a collector. This is the same pair that appears in SSD flash-translation layers, LSM-tree compaction, table-format metadata and multi-version concurrency control (version chains): a lookup structure that lets data move, plus a cleaner that reclaims what is dead. Here the collector is literally called a garbage collector. Notice what going off-heap does: it does not escape the pattern, it re-creates it at coarser , with your pool as the indirection and your free logic as the collector. The uncomfortable form of the lesson:

One-line compression: off-heap does not remove memory management. It changes liveness from proved-by-traversal to known-by-construction, and it is worth it exactly when the live set is large, uniform and long-lived.