Pools, max active runs and depends_on_past

Concurrency controls: pools, limits and depends_on_past

These are three different mechanisms with three different purposes, and they are usually taught as one.

Pools

A pool is a semaphore over a resource that the orchestrator does not own: a warehouse's concurrency slots, a database's connection count, a third-party API's rate limit. The orchestrator keeps a model of a resource it cannot observe. So the limit is a guess, and a stale one. It does not know who else uses the warehouse, or that the database administrator halved max_connections last week.

Set too low: capacity sits idle.
Set too high: your pipelines become the load that overloads your own warehouse, and because the orchestrator retries, the overload feeds itself.

This is admission control (limiting how much work is allowed in) done by a party that has no feedback signal. The principled version uses backpressure from the target (the target tells you to slow down) instead of a static number. Almost nothing does this, because most targets expose no backpressure signal. They give only errors, and errors cannot be told apart from failures.

Max active runs

Max active runs limits run-level concurrency for one workflow. Its real job is to contain catchup and backfill herds (Safe re-execution) and to cover cron's missing assumption that the previous run finished (Graphs and time). Setting it to 1 is the common, correct, boring choice for daily batch work.

depends_on_past

depends_on_past is different in kind. It says run N may not start until run N−1 has succeeded, which serializes the sequence of runs. It exists for cumulative state: running totals, slowly changing dimension chains, anything where the interval is not independent.

The cost is exact. You turn a parallelizable backfill into a sequential one, so backfill duration = number of intervals × per-interval runtime, and no amount of capacity helps. The second cost is worse: one failed historical interval blocks everything after it indefinitely. A transient failure on day 3 stops days 4 to 30, and recovery is manual. Use it only when the dependency is truly cumulative. Prefer to restructure the computation to be interval-independent, because interval independence is the property that makes every other mechanism here cheap.

Pools are admission control without a feedback signal; depends_on_past buys ordering and pays for it in backfill wall-clock and in blocked successors.

Case, resolvedConstructed case · Tomás at Marigold Market

A renamed folder, and a sensor that never fires

The refund team renamed its output folder and Marigold's pipeline waited on the old path silently, and at month end every team's pool together overloaded the shared warehouse. Where it started: Teams and limits →

The renamed folder. Tomás and the refund team's lead agree on a contract. The table is called refunds.daily; the name is registered in a shared catalog with an owner; any path change must be announced two weeks ahead; and the refund pipeline publishes a dataset event when each partition is ready. Tomás's workflow now declares refunds.daily as an input instead of watching a folder. He keeps a slow poll, once per hour, as the "braces" in case an event is lost.

The month-end overload. Each pipeline had a pool of 40 slots in the orchestrator's model, but the warehouse really gives his team 20. Tomás sets the pool to 12 (60% of his team's share, leaving room for retries and other users), sets max active runs to 1 for each daily workflow, and removes depends_on_past from every pipeline where days are independent. Only the running-balance table, which really is cumulative, keeps it.