Scenarios

Past the knee of the curve, more concurrent work adds latency but no throughput. Reloads add load, and the queue holds requests whose users have already left, so throughput looks high while goodput is low.

1. A bounded queue, sized as depth = tolerable wait × service rate: 5 seconds at 100 per second is 500.
2. A deadline on every request, checked when it is dequeued.
3. An AIMD concurrency limit.
4. Fast rejection with "retry in a few seconds", which breaks the reload loop.

Some users are refused, but those admitted get answers. A limit set too low wastes capacity, which is why AIMD keeps probing upwards.

It is a gang job: it needs all 64 machines at once, and the free machines are scattered or keep being taken by short jobs.

A reservation that holds machines for the gang as they free up, plus backfilling short jobs, with enforced time limits, into the gap while it fills.

Machines sit idle while the gang accumulates: the price of making the job possible. Backfill works only if users declare honest time limits, so enforce or price them. Allocate gangs atomically, so two half-allocated gangs cannot deadlock.

It survives if any replica does, so spread replicas across racks and zones. With a 1% chance per day of a rack failing, one rack risks 1% downtime; three racks, 0.0001%.

It fails if any task fails, so concentrate it in one rack: spreading a gang over 4 racks raises its failure chance from 1% to 3.9%, and adds cross-zone bandwidth cost.

Spreading lowers packing density and needs a descheduler to correct drift. Keep shuffle data on local NVMe.

The borrowed tasks have a p99 duration of about 55 minutes, and without preemption the slots return only as those tasks finish.

Cap task length on borrowed capacity at, say, 10 minutes, so reclaiming takes at most about 10 minutes; allow preemption of borrowers, which loses their unsaved progress; or stop lending, which leaves capacity idle.

A task-length cap plus preemption, with checkpointing for long borrowers.

Guarantees, utilization and no wasted work cannot all be had at once.

For 10,000 nodes, 30-second tasks need about 300 placements per second; 100 ms tasks need 100,000.

If the scheduler manages 5,000 per second, batch tasks into units of about 5 seconds (50× less demand), and add a sampling fast path (two probes, shorter queue) for tiny tasks, keeping the careful scheduler for gangs, quotas and fairness. A shared-state scheduler is another option, but conflicts and retries grow as the cluster fills.

The claim that monolithic schedulers do not scale is cited more often than demonstrated, so measure before redesigning.

The burst is shorter than the scaling loop's delay, so scaling can never help, and the loop flaps.

Absorb spikes with headroom sized for the known peak, a scheduled scale-up shortly before it (with a forecast biased high, since under-provisioning costs more), a bounded queue, and shedding. Scale to zero only if the gap between requests is much longer than the warm-up time.

Headroom is money spent on idle capacity.

With a mean of 10 slots and a standard deviation of 5 per tenant, planning for mean + 2 standard deviations: separate clusters need 16 × 20 = 320 slots; a shared one needs 160 + 2 × 20 = 200, 37.5% less.

That gain assumes tenants are independent, but demand correlates at peak: if all of them peak at 20 together, the shared cluster needs 320 too.

Shared pools for tenants whose peaks do not coincide, separate pools for heavy or correlated ones, and price isolation as the multiplexing gain it gives up. Reserve a short-query lane, for example 10% of capacity, so short queries are not stuck behind long ones.

Head-of-line blocking: one first-in, first-out queue carrying work of very different costs.

A bulk job of 5,000 rides at 2 ms each holds a worker for 10 seconds, while 300 small requests per second pile up behind it: 3,000 in those 10 seconds.

Split bulk work into 50-ride chunks (100 ms each), and give bulk work its own pool. Alert on the age of the oldest waiting request.

Separate pools lose some multiplexing efficiency and add configuration. Then check the layer below for the next shared queue.