Cost-based workload routing

How should queries be routed to engines, and which way should the router err?

Route the query to the cheapest engine that can serve it: small queries to a small warehouse or a single-node engine, large scans to the distributed one. This is admission control with a destination. It needs the same missing number (how big is this query?), with the same escape: route on cheap early features (estimated bytes, partition count, number of joins) rather than a true cost model. And bias the classifier toward the failure you can afford:

Misrouting big work to a small engine fails loudly and late.
Misrouting small work to a big engine merely overpays.

Bias toward the big engine, and the mistake becomes recoverable.

Route on cheap evidence, and bias the classifier toward the failure mode you can afford.

Case, resolvedConstructed case · Lena at Brightlane

Monday 09:00: reloads, abandoned queries, late machines

On Monday at 09:00 dashboards slowed past a minute and users reloaded, abandoned queries kept running, the autoscaler arrived 4 minutes late, a runaway query failed at 95%, and a team within quota slowed everyone. Where it started: Workload management →

Slow dashboards and reloads. She sets a bounded queue (about 5 seconds' worth: depth = 5 s × service rate) and an adaptive concurrency limit (AIMD on latency). Past the limit, Hub says "busy, retry in a few seconds" immediately. This breaks the reload loop, because a fast "no" costs almost nothing.
Abandoned work. Every request now carries a deadline, and workers check it at dequeue. Goodput rises: the cluster stops spending capacity on answers nobody will read.
Autoscaling. Burst length (about 90 s) is shorter than the loop delay (4 minutes), so scaling cannot absorb it. She adds headroom for the 09:00 peak, a scheduled scale-up at 08:50 (a forecast, biased high), and load shedding for the rest.
The hour-long runaway query. She adds a kill rule based on early evidence: estimated bytes scanned over a limit are rejected at admission, and a query whose memory growth rate is extreme is stopped within seconds, not at hour one.
Noisy neighbor. The heavy team was within quota, but contended on cache and memory bandwidth, which are outside the quota namespace. She gives that team its own pool on separate machines (space-sharing) and prices it as the lost multiplexing gain.
Short queries. She creates a reserved short lane (10% of capacity), routes by cheap features (estimated bytes, number of joins), and biases the classifier toward the short lane when unsure.