Scheduler self-durability and scale

What limits a scheduler's loop, and why is its metadata database the real single point of failure?

The control plane is itself a system. The repeated failure to treat it as one is why "the orchestrator went down" is a whole category of incident.

The loop and its cost

The scheduler's loop is: parse definitions → evaluate readiness → write state → hand to executor. The cost is parse time × number of definitions × parse frequency, and that product sets a floor on scheduling latency.

When definitions do I/O at parse time (as in dynamic fan-out), the floor rises with data volume rather than with code volume, and every workflow slows down together. The practical rule is: definition files must be cheap and free of side effects to evaluate. It is the rule most consistently broken by people who have just discovered dynamic generation.

The architecture is forced

Durability forces the architecture:

Run state must survive scheduler restarts, so it lives in a database.
Several schedulers must not launch the same task twice, so they need mutual exclusion.
Mutual exclusion needs a database with transactions and row-level locking, and often a leader election on top (the schedulers choose one that is in charge).

The workload (small writes, high contention, strict transactional need, tiny volume) is a textbook relational workload. This is why essentially every orchestrator ended up with Postgres or an equivalent plus a leader. That agreement is the evidence: the shape of the problem selected the storage engine, not fashion.

What people under-plan

The metadata database is a single point of failure for every pipeline at once, and it is sized by task count, not data volume. A dynamic fan-out and a chatty heartbeat interval can make a control plane that serves ten terabytes of pipelines collapse under the write load of its own bookkeeping. And this mechanism's own state is an append-only run history, so it again needs a retention policy and a cleaner.

The scheduler is a small, high-contention transactional system whose database is sized by task count; parse cost sets the scheduling latency floor and definitions that do IO raise it for everyone.

Case, resolvedConstructed case · Tomás at Marigold Market

Top priority, and still 25 minutes late

Marking the finance close pipeline as top priority changed nothing, and the scheduler itself grew slow as definitions multiplied. Where it started: Priority and the scheduler →

The decorative priority. Tomás measures queue time in the warehouse and finds that 25 of the 28 minutes of delay are there. He stops relying on orchestrator weights. Instead he creates a reserved warehouse queue for finance close (the warehouse has its own workload classes), and passes the queue name from the task configuration so the queries enter it. The backfill gets its own pool of 2 slots, so it is never starved and never competes with production.

The slow scheduler. He finds three definition files that list an object-store prefix at parse time. He moves the listing into a task that runs once per day, so parse is cheap again. He adds a retention job that deletes run history older than 90 days, sets heartbeat intervals to 30 seconds instead of 2, and runs two scheduler copies with leader election. The metadata database's connection count and write rate now go on the dashboard, next to the pipelines.