Summary and key takeaways
The one sentence. Orchestration is the discipline of making unreliable, unobservable, irreversible side effects behave like a pure function of a time interval.
What is different from lower layers. The control plane moves no data and costs three to six orders of magnitude less than the work it commands. So coordination is free and its correctness is everything. Three constraints replace performance: the coordinator cannot observe effects, cannot undo them, and must keep state across durations longer than any process. Time becomes an argument: the interval key is an address.
The six impossibilities, one line each:
- You cannot verify success: exit code 0 is a claim. Build independent evidence (assertions, write-audit-publish).
- You cannot undo effects: recovery is forward-only, by idempotent re-execution.
- You cannot tell dead from slow: fence relaunches; at-least-once plus idempotency is the real guarantee.
- You cannot detect completeness from arrivals: use a sentinel, a count or a timeout.
- You cannot detect your own absence: use an external dead-man's switch.
- You cannot verify a declared dependency: undeclared edges cause the incidents.
Key equations and rules:
| Rule | Meaning | Seen in |
|---|---|---|
| L = λW (Little's Law) | work in the system = arrival rate × time inside | sensors holding slots; pool limits |
| Run identity = (version, interval, params) | anything not pinned to the run makes replay a lie | parameterized runs, backfill |
| Detection latency ≈ ½ poll interval; poll cost = (wait ÷ interval) × per-poll cost | polling trade | sensors |
| Useful share = work ÷ (work + launch cost) | fan-out is a batch-size decision | dynamic task generation |
| Backfill time = intervals × per-run time (if ordered) | the price of ordering | depends_on_past, cumulative state |
Patterns that repeat:
- Waiting must not hold the resource being waited for: sensors, sub-DAGs and approvals all follow from this.
- Idempotency needs identity and determinism, and costs write amplification.
- Atomicity is borrowed from the storage layer (pointer swap, fencing tokens), never owned by the orchestrator.
- Choreography moves global state to tracing; orchestrate inside a team, use contracts across teams.
- Skipped is a third value, and default join rules turn it into a green run with no output.
- Dependency graphs are cheap, dependency contracts are not.
- Priority set upstream is advisory: the last queue's policy wins.
- The scheduler is a transactional system sized by task count.