Skip to content
About
Main content

Choose a coordination model

The decision

Several steps must run in order: extract, clean, load and refresh a dashboard; or validate, approve, refund and notify a customer. Either one component knows the whole plan and drives it, or each step reacts to an event from the one before. And a coordinator can run each step as a separate task, or run the whole process as one program whose progress is logged. Which fits this work?

You will end with a coordination model for the work, and what it costs in visibility, retries and versioning.


Decide

1

Do all the steps belong to one team?

One team owning extract, clean and load; versus a producer team publishing data that other teams' pipelines consume.

Why

Inside a team, coordination is free and a global view is valuable. Across an ownership boundary, every change to a shared central definition is a request to another team, so the coupling is what costs. The usual answer is orchestrated islands, connected by events.

2

Is the state passed between steps large, and the control flow simple?

A hundred-gigabyte table passed along a straight line of steps; versus small state with loops, conditions, waits for approval or compensations.

Why

A graph of tasks keeps state outside the , in storage, so it suits bulk data where the state could never fit in a program's variables. keeps state in the program's own variables and logs every external call, so it suits long-lived processes with complicated control flow and small state.


Where you can land

Chain them with events between teams

Use it when the steps cross an ownership boundary.

Each stage publishes an event ("orders export finished for 5 March") and downstream stages react. A new consumer needs no change to the producer.

What it costs: no component can answer "is this run done, and if not, where is it?". The global picture does not disappear; it moves into the observability system, reconstructed after the fact and only approximately. Attach a to every event of a run so it can be.

A graph of tasks (DAG)

Use it when one team owns the steps, the state between them is large, and the flow is simple.

Each task is a separate process; the orchestrator records the status of every task per run, so "where is Saturday's run?" is a lookup.

What it costs: the unit of is a whole task, chosen by its author; steps cost minutes to launch, so they cannot be small; loops and conditions are awkward; and a changed definition can silently mean something different for old runs.

Durable execution

Use it when one team owns the steps, the state is small, and the control flow is complicated.

The process is ordinary code. Each call to an external system has its result logged; after a crash, the code from the start and logged calls return their recorded results.

Before you build it

Make the workflow code deterministic

Replay must take the same path as the original run. Branching on the current time, a random value or a direct network call breaks it; the runtime must supply each of these through logged calls.

Plan for versioning

Changing the code while old runs are in flight makes replays diverge, and that divergence must be handled explicitly in code.