Orchestration & Workflow
How do you make unreliable, unobservable, irreversible side effects behave like a pure function of a time interval?
Who this mechanism is for: a developer who writes code, schedules jobs with cron or has seen a tool such as Airflow, and knows basic SQL, but has never studied how a workflow scheduler really works inside. You do not need any database or distributed-systems background. Every technical term is explained the first time it appears, and again in the glossary at the end.
The story we will follow
Tomás is a data engineer at Marigold Market, an online grocery company. Every night a pipeline builds the daily_sales table. It copies the day's orders out of the shop database, cleans them with a Spark job (a tool that processes large data on many machines), loads them into a cloud data warehouse, and finally refreshes the dashboard that the finance lead, Nadia, reads at 07:00. Other teams also depend on these tables: the marketing team, the refund team and a partner company that sends delivery files.
At the start the pipeline is a handful of cron entries (scheduled commands). During the next quarter Tomás will move it to a workflow orchestrator and meet eight problems, one at the start of each article below. Each problem looks like a bug. Each one is actually a structural limit of orchestration: a thing that the coordinator cannot know, cannot undo, or cannot wait for. At the end of each article we go back and solve the problem with that part's mechanisms.
This is the main idea of this whole mechanism:
How this mechanism is organized
- 1
A layer that moves nothing
What is different about a layer that moves no data, and who holds the global picture: one coordinator or many reacting parts?
CaseIs Saturday's run finished? Nobody knows
- 2
Graphs and time
What does a dependency arrow really mean, and what does "the run for the 5th" mean?
CaseA green run, and Sunday's numbers on Monday
- 3
Triggers
How does work become ready: by clock, by event, by polling, or by dataset name?
CaseEvery worker busy, and nothing running
- 4
Safe re-execution
How do you replay history, retry a task and survive a task that is dead or only slow?
Case730 runs at once, and a torn partition
- 5
Deadlines, branches, graph shape
How do you detect work that never started, and what do branching, fan-out and grouping cost?
CaseGreen for 60 hours, and nothing ran
- 6
Teams and limits
Why are cross-team dependencies hard, and how do you limit concurrency against resources you cannot see?
CaseA renamed folder, and a sensor that never fires
- 7
Durable execution
When is a program that logs its own steps better than a graph of tasks, and what is a run's identity?
CaseAn approval lost in a restart
- 8
Priority and the scheduler
Why does priority often do nothing, and how does the scheduler itself fail?
CaseTop priority, and still 25 minutes late
- 9
What orchestration forbids
What does this mechanism forbid, what escapes exist and what do they cost?
CaseA quarter later: one row answers Nadia
After that: a summary, hands-on labs, interview questions and a glossary.
One request: read the one-sentence ending of each section (the quoted line in italics). Each is the whole lesson of its section in short form.
What do I do, and what does each option cost?
Can I see it myself?
The sensor deadlock
Twenty sensors hold the slots of a 10-slot worker pool while they wait for processing tasks that need a slot.
Break itPythonFind the cost of ordering
60 one-second intervals run sequentially, on 4 workers, and on 4 workers with depends_on_past; then interval 10 fails.
MeasurePythonIdempotent versus append retries
A daily load crashes halfway and is retried: append with and without a transaction, overwrite of the partition, and overwrite with non-deterministic values.
Break itPythonCheck that your dead-man's switch really fires
A scheduler that hangs: a heartbeat checker outside it against the scheduler's own SLA alert.
Break itPythonFan-out as a batch-size decision
Launch overhead of a subprocess and of a container, the useful share of tasks from 1 s to 600 s, and 1,000 tiny tasks one by one against in 10 batches.
MeasurePython, DockerReplay a durable log
A 40-line durable-execution runtime: kill a workflow after two activities, replay it from its log, then branch on the clock and watch the replay diverge.
Break itPython
Can I reason about it under questioning?
Interview questions
Short questions on workflow orchestration, such as DAGs and data intervals, sensors and triggers, backfills and safe retries, deadlines and branching, pools and priority, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on workflow orchestration, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Absence is not evidence
Absence of an event is not absence of a failure.
Designate one write
Atomicity over n items is atomicity over one designated item.
Little's Law
Concurrency = Throughput × Latency.
Costs are conserved
Costs are conserved, never removed.
Variance costs more than volume
Variance costs more than volume.
Show all 11 principlesShow fewer
Granularity mismatch
Granularity mismatch creates amplification.
Independence is assumed
Independence is assumed everywhere and is true almost nowhere.
Fixed cost per operation
Fixed cost per operation implies a natural batch size.
The end-to-end argument
Guarantees compose to the weakest link, and only endpoint guarantees are real.
Hidden state breaks replay
A replayable pipeline is a pure function of its retained inputs.
Undeclared dependencies
An optimization that succeeds becomes an undeclared dependency, with leverage 1/(1 − h).