Orchestration moves nothing

Why is coordination free here, and what can an orchestrator never observe or undo?

CaseConstructed case · Tomás at Marigold Market

Is Saturday's run finished? Nobody knows

Nadia asks a simple question on Monday morning: "Is the pipeline for Saturday finished, and if not, where is it?" Tomás cannot answer. Marigold's pipeline today is a chain of reactions. When the order export finishes, it drops a file. A watcher on that file starts the Spark job. When Spark finishes, it posts a message. A third script listens for that message and loads the warehouse. Nobody owns the whole chain. To find the answer Tomás reads logs in four systems for forty minutes. Then he finds that one step ended with exit code 0, but it processed an empty file.

Two questions are hidden here. Who knows where the pipeline is? And what does "finished" even mean? This article answers both.

How it ends →

Everything below an orchestrator in a data system moves data: disks read it, networks carry it, query engines transform it. An orchestrator moves no data at all. It sends commands into systems it does not own, and then it waits. The arrows in its graph do not carry bytes. They carry claims: "the thing that was supposed to happen, happened." The data itself travels out of band, through storage that the orchestrator never reads.

Coordination becomes free

That fact reverses the economics. In a database or a query engine, the coordination machinery sits on the critical path of every byte. Its cost is measured against data volume, and the whole craft is to make the coordinator cheap enough to disappear. In an orchestrator, the control plane costs three to six orders of magnitude less than the work it commands.

So for the first time, coordination overhead is free and the correctness of coordination is everything. Nothing here is about throughput. Cache lines, vectorization and serialization width all stop mattering. (The one exception is the scheduler's own bookkeeping load, Priority and the scheduler.)

Three things the orchestrator cannot do

What takes the place of performance is a constraint that earlier layers of a data system never face:

Compare with a query engine. An engine owns its operators' memory. It can kill a task and recompute it exactly from a recorded recipe (lineage: the record of how each result was computed from its inputs), because it controls both the input and the state. An orchestrator controls neither. It launches a process in a container on a cluster it does not manage. That process writes to a store the orchestrator does not read. It then receives back one integer.

Exit code 0 means the process believed it succeeded. It does not mean that the partition is complete, or correct, or written once. Every mechanism here is a workaround for this gap.

Duration: state must outlive everything

The second new constraint is duration. A query lives seconds to hours, and nothing in it outlives its process. A workflow spans intervals longer than any process, any machine, any deployment of the orchestrator itself, and often longer than the code that defines it. So all state must be moved out of memory into a durable log of what ran. The orchestrator's own database is therefore a write-ahead log in different clothes: it records what was started and what finished, so it can recover after a crash. Its run history is append-only (new records are added, old ones never edited), and like every append-only store it needs a retention policy and a cleaner that deletes old entries, or it grows forever.

Time becomes an argument

The third constraint: time becomes an argument, not a cost. In lower layers, time is something you spend (latency). Here, the logical interval is a parameter passed into the work. It is the key that makes a run addressable, repeatable and comparable to the run before it. A daily job does not mean "run at 02:00"; it means "process the slice of data for 5 March." Graphs and time develops this.

The failure unit is chosen by a human

One more structural consequence is worth holding onto. This is the first layer where the failure unit is chosen by a person. A task boundary decides four independent things at once:

the retry granularity (what is re-run after a failure),
the checkpoint granularity (what progress is saved),
the parallelism granularity (what runs side by side),
the observability granularity (what you can see in the UI).

Four separate concerns collapse into one arbitrary decision. It is made once, usually without much thought, and it is expensive to change later.

A pure function always gives the same output for the same input and has no side effects. That is the target. The rest of this mechanism shows what it takes to reach it.