Columnar memory pooling (Arrow-style buffers)

Why does pooled, typed, reference-counted memory make a footprint measurable, and what does sharing cost?

The constraint

In a columnar engine (one that stores each column's values together) the unit of work is a batch of a column: a contiguous run of fixed-width values plus a validity bitmap (one bit per value: is it null or not?). This unit has three properties that nothing else in this article has:

1. Its size is computable before allocation.
2. Its lifetime is shared across the operators of a pipeline.
3. Its alignment is required: 64 bytes, so that SIMD loads (one instruction on many values) do not straddle two cache lines.

Therefore: typed pools with accounting

Allocation can be pooled and typed rather than general. A memory pool hands out aligned, padded buffers and, critically, tracks the total bytes outstanding. That accounting is not a diagnostic extra. It is the precondition for everything in Rationing: you cannot grant, reserve or admit against a footprint you cannot measure, and you cannot measure a footprint that is spread across a general-purpose heap in objects of unknown origin. Pooling is what makes memory a governable resource at all.

Therefore: reference counting and zero-copy

Buffers are reference-counted so that columns can be shared without copying. A filter that produces a selection vector (a list of the row numbers that passed) does not copy the data column. It keeps the buffer and adds a reference. A projection that drops columns simply releases them.

The deep point

Arrow's layout is chosen so that the in-memory representation and the wire representation are byte-identical. That merges the serializer and the memory manager into one component. Deserialization allocation, routinely the largest allocation source in a data system, goes to zero, because you point at bytes you already have. Data crossing a process, language or client boundary never touches the allocator. The choice of encoding belongs to a storage-format discussion; the consequence belongs here: the choice eliminated an allocator.

Costs, honestly

• Atomic reference-count increments on shared buffers are writes to cache lines that many cores want. This is cache-line contention (cores fighting over one cache line) arriving inside the memory manager, and the same coherency cost that makes scalability curves bend downward.
• Zero-copy sharing couples lifetimes. One slow consumer holding one reference pins the whole buffer. A straggler keeps a batch's memory long after the pipeline moved on. This is the assumption the design cannot verify (that consumers release promptly), and the party that pays is every other operator competing for the pool.
• Padding is granularity mismatch again. 64-byte alignment plus padding on a narrow or short column is mostly padding. Example: a column batch of 10 four-byte values is 40 bytes of data in a 64-byte aligned buffer, so 24 bytes (37.5%) is padding.

One-line compression: pooled, aligned, reference-counted buffers make allocation predictable and footprint measurable. The price is atomic contention and memory pinned by the slowest consumer.