Cross-DAG and cross-system dependencies
Why is a dependency across an ownership boundary an organizational problem rather than a technical one?
Within one workflow, a dependency is a declaration you control. Across an ownership boundary it is not, and the binding constraint stops being technical.
The mechanisms are the three from Triggers, now judged by a different cost:
| Mechanism | Coupling direction | Breaks when |
|---|---|---|
| Consumer polls the producer's output | Consumer → the producer's output layout | They change the path, partitioning or completion convention |
| Producer triggers consumer | Producer → the consumer's existence | They forget, or a notification is lost |
| Both declare a dataset name | Both → a shared namespace | The namespace has no owner |
Polling is the default not because it is good but because it needs nothing from the other team. The consumer can do it alone. It is late (a poll interval behind), it holds resources, and it couples you to an implementation detail (their file layout) rather than to a contract. Triggering needs the producer to know and maintain a list of consumers, which they will not do. It fails by omission, silently, when a new consumer appears or an old one moves. Shared dataset declarations are the only shape that couples both parties to a contract instead of to each other, and they need a catalog that both teams agree to use. That is an organizational commitment, not a deployment.
The most important point of the layer
The technical cost of a cross-team dependency is negligible: a name, a poll, a webhook. The expensive part is the agreement about what the name means, who may change it and what notice they must give.
Where the org chart has a boundary, the DAG grows a sensor, and that sensor is a workaround for a conversation that did not happen. This is Conway's law (organizations produce designs that copy their own communication structure) expressed in polling intervals.
This also settles the first fork of Orchestration moves nothing. Inside a team, orchestrate: coordination is free and the global view is valuable. Across teams, use declared datasets or events: the coupling is what costs.
Across an ownership boundary the constraint is organizational — the sensor is the technical residue of an absent contract.