Prioritization and short-query lanes

Why does priority alone not speed up short queries, and how should a classifier err?

The thing people get wrong: priority ordering alone does not accelerate short queries. Ordering applies at the queue. A short query that has already been admitted behind a long one in the same resource pool waits for the long one, whatever its priority. To fix this you need one of three things:

1. Preemption (stop the long query);
2. A reserved lane whose capacity the long queries cannot touch;
3. Classification at admission that routes the two kinds apart.

Reserved lanes are the common answer. Their price is idle: the lane must stay empty when no short queries exist, or it is not reserved.

This is isolation versus utilization at its plainest:

Classification and the asymmetry of errors

Classification hits this mechanism's central constraint: you are guessing duration again. The saving grace is asymmetry.

• Misrouting a long query into the short lane costs one lane's capacity for one query.
• Misrouting a short query into the long lane costs that query its latency.

The two errors are not equally expensive, so the classifier should be biased, not merely accurate: when unsure, send the query to the short lane.

Priority reorders a queue. Only a reserved lane or preemption changes who is ahead of you in service.