Scenarios

If each of 300 pipelines holds a slot through an average wait of 4 hours, many hundreds of slots sit occupied by waiting (Little's Law). No sensible pool covers that.

Deferrable sensors that release their slots and register with one shared poller checking every 5 minutes (an average detection delay of 2.5 minutes). Ask the partner for a success marker and a manifest with the expected file count. Add a timeout, say at 14:00, that raises an alert rather than guessing.

The shared poller makes 12 checks an hour × 8 hours = 96 listings a night, instead of 300 × 240 = 72,000.

The shared poller is a new single point of failure, so run two. A partner that forgets the marker is caught by the timeout alert. Add an external dead-man's switch on the final publish.

The task appended. A retry after a partial append leaves duplicates, with no key to remove them by.

Address the output by the interval: replace the append with an overwrite of the day (delete the partition and insert, in one transaction) or a merge on the business key. Ban now() and unversioned lookups. Add write–audit–publish: write to a staging path named after the attempt, assert row count and key uniqueness, then publish with one atomic pointer swap, so a crash leaves nothing visible.

Rewriting a whole partition amplifies writes by the ratio of partition to rows changed, plus a staging copy and the time to run the assertions.

In CI, kill the task at random points and check that the final table equals one clean run.

A run that never starts emits no failure, and SLAs inside the orchestrator are down along with it.

An external dead-man's switch: the last task of each critical pipeline pings an independent service, on another machine with another alert channel, which expects the ping by a deadline (say 06:30) and pages when it is missing. Check the data, that the latest partition is yesterday's, not just the task.

A staleness banner, or serving the last good partition.

Run the scheduler as a highly available pair with leader election. The monitor is a second system with its own availability, so keep it very simple, and have it send its own heartbeat to a third place.

You cannot tell a dead worker from a slow one.

1. Assume dead and relaunch: two writers may tear the partition.
2. Assume alive and wait: the pipeline may stall indefinitely, with no alert.
3. Fence: each attempt carries an increasing token, and the sink rejects stale ones. That needs the sink's cooperation, which object stores and JDBC targets usually do not give.

At-least-once launch; a timeout based on the task's distribution of durations; and write–audit–publish with a compare-and-swap on the table pointer, using the attempt number as the token, so a stale attempt loses at publish time even if raw writes cannot be fenced.

Exactly-once orchestration is not available.

The technical cost is tiny; the contract between teams is what is expensive.

Replace polling of output paths with dataset declarations in a shared catalog: each table has an owner, a name, a convention for signalling completion, and a rule for notice of changes (say 14 days). Producers publish an event when a partition is ready; consumers declare the dataset names they read, keep an hourly poll as a fallback for lost events, and alert externally on freshness.

A shared namespace needs an owner and a process, not just a deployment. Dataset declarations can be coarser than the real partition dependency and cause spurious wake-ups, so declare datasets per partition where the store allows. Declarations are still unvalidated, so capture lineage at run time to find undeclared reads.

Orchestrate inside each team; use events and datasets between teams.

If the table carries cumulative state, the backfill must run in order: 730 intervals × 10 minutes = 7,300 minutes, about 5 days, and no amount of capacity helps.

Run them in parallel in a separate pool, bounded by what the warehouse can spare: with 2 slots, 730 ÷ 2 × 10 = 3,650 minutes, about 61 hours; with 8 slots, about 15 hours.

Turn off automatic catch-up, so the backfill is an explicit action. Overwrite idempotently by key, with write–audit–publish per partition, so a failure leaves the old data visible. Stamp every rewritten partition with the new code version, so consumers can see where old and new logic mix.

Until the backfill finishes, rows from the new code mean something different from the old rows.

The state between steps is small, and the control flow has waits, loops and compensation, which suits durable execution. A task DAG would hold a slot during the human wait (and a deployment will kill that process long before the manager answers), has no natural compensation, and retries coarsely.

A durable-execution engine: the wait is a row, not a running process; each activity retries on its own; the compensating refund is another activity in the log. Add a timeout with a stated default (fail closed, for payments), an immutable approval record (who, when, which version of the request), and pinned parameters.

Replay requires deterministic workflow code (no direct now() or network calls), and code changes need explicit versioning.

Bulk data pipelines as DAGs: their state cannot live in workflow variables.

The delay is downstream. Measure time spent queued in the warehouse or Spark cluster against time in the orchestrator. If, say, 25 of 28 minutes are spent in the engine's queue, orchestrator priority is decorative, since it only governs admission.

A reserved queue or capacity class in the engine for this pipeline, with a way for the task to name its queue; separate pools in the orchestrator so backfills are not starved; and fewer low-priority queries running during the critical window.

Pool limits set too low, a long wait in an upstream sensor, or a delay in the scheduler parsing workflow files.

Reserved capacity lowers utilization, since it sits idle outside the critical window: the price of isolation.