NUMA awareness

Why does the thread that first touches a buffer decide its NUMA node, and how large is the effect really?

The penalty of remote memory access is physics: a remote read costs about 1.5–2.2× the latency of a local one, and the interconnect between sockets has less bandwidth. The question here is placement policy. The mechanism that decides placement is first touch: Linux puts a physical page on the node of the thread that first writes to it, not on the node of the thread that called malloc.

Therefore the allocation path does not determine placement. The initialization path does. A single-threaded loop that zeroes the buffer pool at start-up places the entire pool on one node. Every worker on every other socket then pays remote access for the life of the process. This misconfiguration is common, permanent once made, and invisible in every metric except IPC (instructions per cycle) and memory-stall counters.

// BAD: one thread touches every page first, so ALL pages land on thread 0's node.
memset(pool, 0, pool_bytes);

// BETTER: each worker initializes the slice it will later process,
// so every slice lands on that worker's own node (first touch).
#pragma omp parallel for schedule(static)
for (size_t i = 0; i < n_chunks; i++)
    memset(pool + i * CHUNK, 0, CHUNK);   // same split as the processing loop

The forced designs

Partition the pool per node and bind workers to the node that holds their partition. Use per-node free lists instead of one global list.
Allocate and initialize working memory on the thread that will consume it. This is the underlying reason that per-worker arenas and per-thread pools are the norm, in addition to avoiding allocator contention.
Interleave pages across nodes (the --interleave option of numactl) when access is truly uniform. This accepts a worse mean latency to remove bimodality (two very different speeds depending on page location). It is a variance-reduction choice, not a performance one: predictability is bought with average-case work.
Once memory is bound, thread migration becomes a correctness-of-performance issue, so NUMA-aware memory forces NUMA-aware thread scheduling. That coupling is real. It belongs to execution and scheduling; we name it and stop.

Claim to test

The commonly quoted "~2×" is a latency ratio for remote access. It is routinely repeated as if it were the throughput or end-to-end ratio. For bandwidth-bound scan work on typical two-socket machines the observed effect is often 10–30%. Modern chiplet designs put NUMA-like boundaries inside a socket, so the topology no longer maps to sockets at all. This is current engineering reality, dependent on hardware generation, not a permanent constraint. Measure your machine; do not inherit the number.

A conflict between mechanisms

NUMA binding assumes threads stay where you put them and access is node-local. Work stealing (idle threads take tasks from busy threads) violates the first assumption by design. So two mechanisms from neighbouring parts of an engine are optimizing against each other, and neither one's metrics show it.

One-line compression: placement is decided by first touch, not by malloc. Whoever initializes a buffer owns its node for the life of the process.

Case, resolvedConstructed case · Amara at Voltara

40-second freezes, and a process that looks like it leaks

A 60 GB in-heap metadata cache froze the dashboard for 20 to 40 seconds, RSS grew for days while the heap stayed flat, a global table of interned URLs ran the nightly job out of memory, and one of two identical servers scanned 30% slower. Where it started: Ownership →

The 40-second freezes. The 60 GB meter cache is large, uniform (fixed-size records) and long-lived. All three conditions hold, so she moves it off-heap into a few large buffers with an explicit reference-counted lifetime. The collector now traces a handful of objects instead of hundreds of millions. She keeps the rest of the engine on-heap and tunes only the young generation, because the remaining objects are small and short-lived.
RSS grows while the heap is flat. This is external fragmentation in the allocator for the per-query scratch data. She introduces an arena per query phase, with the reset point named in code (end of the build phase of each join) and per-tuple scratch space reset at each tuple.
The URL table that eats the machine. The global intern table is replaced by per-batch dictionaries with a cap. When a dictionary passes 64,000 distinct values, the batch falls back to plain strings.
The slow twin. The buffer pool was zeroed by one start-up thread, so all of it sat on socket 0. She initializes the pool in parallel with the same split as the scan workers. The scan on the second socket stops paying remote-access cost.