Interview questions
What an orchestrator is
1Why does coordination overhead hardly matter in an orchestrator, and what matters instead?
The decision to launch a job takes milliseconds and the job runs for hours, so the control plane costs three to six orders of magnitude less than the work it commands. What matters is whether coordination is correct, and the orchestrator cannot observe or undo the effects of what it launches: it gets back one exit code, which says the process believed it succeeded, not that the data is complete, correct or written once.
Read moreCoordination becomes freeThree things the orchestrator cannot do
2What does choosing where one task ends and the next begins decide?
Four things at once: what is re-run after a failure, what progress is saved, what runs side by side, and what you can see in the UI. Four separate concerns collapse into one decision that is made early, often without much thought, and is expensive to change.
3Does choreography, where each stage reacts to the previous stage's events, remove the need for a coordinator's global state?
No, it relocates it. With no component holding the plan, "is this pipeline done?" must be reconstructed by correlating events with a trace ID, after the fact and only approximately. What choreography buys is the removal of a coupling point: a new consumer needs no change to the producer. That is why teams orchestrate within a team and choreograph between teams.
Read moreWhy the global picture must live somewhereWhat choreography does buy
Graphs and time
4Why must a workflow graph be acyclic?
In a cycle, every task waits for another task in it, so none can ever start, and completion cannot be defined. Acyclicity guarantees a topological order exists, so every run can finish.
Read moreWhy acyclic
5What does an arrow between two tasks in a workflow guarantee?
Only ordering: B may start after A finishes. It does not guarantee that B reads what A wrote, and the orchestrator cannot check that.
Read moreWhat an arrow means
6What are the two most common bugs in a workflow's dependencies?
An edge with no data dependency behind it, which needlessly serializes work but is harmless; and a data dependency with no edge, a race that works until the first slow day.
Read moreWhat an arrow means
7Why does a daily job 'for the 5th' run on the 6th?
A run covering an interval cannot start before the interval has closed, so when it runs and which interval it covers are at least one period apart. The run's identity is its interval, not the moment it started.
8How should schedules handle daylight saving time?
Schedule in UTC, and convert to local time only for display. An hourly schedule in local time skips or duplicates an interval on the days the clocks change.
Triggers
9Can an event-driven pipeline know that all of its data has arrived?
Not from arrivals alone, because completeness is an absence: nothing more is coming. You need a sentinel (such as a success marker), an expected count, or a timeout.
10Why can many sensors freeze a scheduler?
Each waiting sensor holds a worker slot, so by Little's Law waiting arrival rate × waiting time worth of slots are occupied by tasks doing nothing, and the tasks they wait for cannot start: a deadlock. Use deferrable sensors, which give up their slot while waiting.
11What does push-based triggering need underneath it?
Triggers that are safe to receive twice, since duplicates happen; and a slow polling fallback, because a lost notification looks exactly like "not ready yet".
Read morePush
12What does dataset-aware scheduling decouple, and what does it cost?
It frees consumers from knowing which workflow produces their input or when it runs. The cost is a declaration of inputs that nothing validates, and a granularity, whole datasets, that is often coarser than the real dependency.
Read moreRebindingTwo limits
Safe re-execution
13Why is a backfill not always a replay?
It is a replay only if the task is a pure function of its interval. Reading now(), reading a table that has since changed, or appending makes it a different computation. And old intervals are re-run with today's code.
Read moreBackfills, catchup and versioningVersioning: the unsolved part
14Why is automatic catch-up a bad default?
It turns unpausing a workflow into a workload of unbounded size: the number of runs launched depends on a start date in a config file, with no confirmation.
Read moreCatchup: a design error
15What makes a task safe to retry?
Output addressed by the interval, written by overwrite, merge or atomic rename, plus a body that computes the same result every time. Appends are never safe to retry.
16Why is exit code 0 not proof that a task succeeded?
It is the task's own claim, made by code that may itself have failed. You need independent evidence: marker files (weak), assertions on the output, or a write–audit–publish flow (strongest).
Read moreCompletion verification
17What is a task's unknown state, and how do you handle it?
Its heartbeats stop, and you cannot tell whether it died or is merely slow. Relaunching risks two writers; waiting risks a stall. The answer is fencing: each attempt carries an increasing token, and the sink rejects stale ones, if the sink supports it.
Read moreThe unknown stateFencing
18Can an orchestrator guarantee exactly-once execution of a task?
No. It guarantees that a task is launched at least once. Exactly-once effects need idempotent tasks and a sink that cooperates.
Read moreThe honest part
19How many retries should a task have?
Not a reflexive three. Enough to cover how long transient failures actually last, with failures that will recur every time routed to a path that does not retry.
Deadlines, branches and graph shape
20How do you detect a pipeline that never ran at all?
With an observer outside the orchestrator: a dead-man's switch that expects a heartbeat by a deadline. SLAs inside the orchestrator fail exactly when the scheduler is down.
Read moreThe run that never startedWhy the orchestrator cannot watch itself
21How can a run finish green and produce nothing?
A branch was skipped, and the join after it uses the default "all upstream succeeded" rule, which treats "skipped" as not succeeded, so the join and everything after it are skipped too. Use a rule such as "none failed" at joins.
Read moreSkipped is a third value
22When is generating tasks dynamically, one per item, a bad idea?
When the work per item is small compared with the cost of launching a task, often tens of seconds. Batch the items, or loop over them inside one task, to spread the fixed cost.
Read moreIt is a batch-size decision
23Why were sub-DAGs deprecated in Airflow?
A parent task waiting for its children occupied a slot that the children themselves needed: a deadlock. Grouping tasks should only be a naming convenience.
Read moreA historical failure
Teams, limits and priority
24Why does making each run depend on the previous one hurt?
It serializes runs, so a backfill takes intervals × time per run, and one failed interval blocks every later one. Use it only for state that is genuinely cumulative.
Read moredepends_on_past
25Why does setting a task's priority high in the orchestrator often do nothing?
The orchestrator's priority only decides when a task is admitted. The task then joins the queue of the engine it calls, whose policy dominates. Measure how long tasks wait in the engine's queue.
Read moreThe last queue wins