Priority and starvation across pipelines
Why does upstream priority rarely change who runs first, and how do backfills starve live work?
Who runs next is the resource scheduler's job. What is specific to orchestration is that the orchestrator's priority governs admission, not resources. A high-priority task jumps the orchestrator's queue and is launched promptly. Then it joins the warehouse's first-in-first-out queue behind forty low-priority queries, where the orchestrator's opinion is not represented.
The last queue wins
The general statement:
Priority set upstream is only advisory unless it is passed on as a token that the downstream queue honors, and almost nothing passes it on: the distributed form of priority inversion. So "we set priority on the critical pipeline" is often folklore: it changed which task was launched first, and nothing after that. The check is simple: measure queue time in the engine, not in the orchestrator. If most of the delay is downstream, your priority setting is decorative.
Starvation
Starvation has a characteristic shape here. Backfills and catchup runs are large, low priority and numerous. Production runs are small, high priority and periodic. With strict priority, the backfill is starved forever: it never completes, and it was probably fixing a correctness problem.
The usual answers apply:
- aging (a task's priority rises the longer it waits),
- reserved capacity per class (a fixed share of slots for each class of work), and
- separate pools.
The choice among them is the general trade between isolation and utilization, with nothing orchestration-specific about it. The only special point is that the workload classes are unusually well separated and easy to name. This makes separate pools the pragmatic default.
Orchestrator priority buys admission order only; in a chain of queues the last policy wins, and reserved capacity beats priority whenever the classes are nameable.