Capacity isolation is not bandwidth isolation

Why can two tenants both fit their memory limits and still ruin each other?

Every rationing mechanism above bounds bytes: a cgroup's memory.max, an executor's heap setting, a container's allocation. All of them count capacity. None of them bounds memory bandwidth (bytes moved per second) or the occupancy of the last-level cache (LLC), the large cache shared by all cores of a socket.

So two tenants can both fit in their limits and still ruin each other. A streaming scan with no data reuse constantly evicts the shared LLC and saturates the memory controller. A co-located latency-sensitive query sees its effective memory latency rise sharply, with no metric changing in either container's accounting. Both are inside their limits. Both are correctly provisioned. One of them is being destroyed. (This is what Amara will find: a big scan job slows the dashboard while every dashboard counter looks normal.)

The numbers have the shape of the bandwidth wall meeting queueing: one socket has a fixed memory bandwidth, a handful of scan threads can approach it, and past saturation latency rises on the familiar 1/(1−ρ) curve. Bandwidth is a really contended resource with a queueing response and, in most data stacks, no gate of any kind.

Mechanisms to control it exist. Intel's RDT (Resource Director Technology) offers cache allocation (it can reserve LLC ways, that is, slices of the cache, for one class of work) and memory bandwidth allocation (it can throttle a class of work). They are almost never used outside the fleets of the very largest cloud companies. Read this honestly: it is current engineering reality and a real gap in mainstream resource managers. It is not folklore, and it is not a permanent constraint.

The structural lesson reframes this whole article: memory is two resources, not one.

Resource	Preemption	Right control
Capacity (bytes held)	not preemptible	ration at admission
Bandwidth (bytes moved per second)	preemptible at every access	should be scheduled continuously, and almost nowhere is

This makes a prediction that can be tested: as scan-heavy work makes bandwidth the binding constraint more often, expect bandwidth scheduling to appear in resource managers, arriving from the isolation side (keeping tenants apart) rather than the performance side.

One-line compression: byte limits bound capacity, not bandwidth. Two tenants that both fit can saturate one memory controller, and nothing in the accounting will ever show it.

Case, resolvedConstructed case · Amara at Voltara

A tiny lookup killed, a giant join spared

On Monday at 9:00 am twelve queries died out of memory, including a tiny lookup while a 100 GB join survived, a stricter cluster queued queries with 60% of RAM idle, and a UDF held memory no counter showed. Where it started: Rationing →

Dmitri's puzzles, and one more, now have answers.

The tiny lookup died and the giant join lived. Without grants, failure strikes the last allocator. Amara adds grants. The giant join must now ask for its memory before it starts, and the lookup is never the victim of a pool a join has already filled.
The idle-memory cluster. She compares the granted number with the actual peak for the top 50 plans. Grants are 8 to 12 times the real peaks, because the optimizer's cardinality estimates for multi-join plans are far too high. She turns on grant feedback with a stability check, quantizes grants to merge-pass boundaries, and adds a per-query cap of 25% of the pool. Queued time drops, because reservations now track real peaks.
The invisible UDF memory. The UDF holds its own hash maps, outside the manager. She sets the reserve's lower bound from measured UDF peaks (about 12 GB across the node) and adds byte-counting wrappers where the UDF API allows them.
Monday correlation. The batch window and Yuki's joins always overlap, so peaks are correlated. She moves the over-commit setting back to plain reservation for the 8:30–10:00 window, and she adds hysteresis (start spilling at 90%, stop at 75%) with a reserve sized to a 3-second spill.
A last discovery. On the dashboard machine, p99 latency doubles whenever Yuki's scan runs, although no memory or CPU counter changes. The scan saturates memory bandwidth. Capacity limits do not see this. She notes it as an open gap and moves the scan jobs to separate machines.