Data movement, stepped back

A concrete pipeline decision is really one choice from each group (transport, routing, flow control, network), and the choices constrain each other.

The constraint chains that run through all four groups

• Choosing a log gives you replay and per-key order. That caps parallelism at N (the number of partitions), which sets your batch sizes, which sets your compression ratio and your file sizes downstream. One decision at the top has consequences four layers down, in your storage bill.
• Choosing pull gives you structural backpressure and natural batching. That gives throughput and costs a latency floor.
• Choosing key partitioning gives you local state and order, and hands you skew, which no amount of flow control fixes.
• Choosing at-least-once delivery, which you did whether you decided it or not, means you owe an idempotency key and a dedup store. The place to answer that is at the sink.

---

What kind of domain this is

Storage is about keeping things. Computation is about deriving things. Movement is the only domain about the gap, and gaps are where systems that were each correct become collectively wrong.

This is not a slogan. It explains the character of the domain: almost none of its hard problems are visible from inside either end. The producer is fine. The consumer is fine. The pipeline is broken. Every debugging story in this domain starts with two teams who each correctly report that their side is healthy.

The consequence: movement is the domain of the un-owned. A record in flight belongs to nobody. It has been sent but not received, acked but not processed, buffered but not durable. Every mechanism here was, underneath, an answer to a custody question: who holds this record right now? Who holds the knowledge that it was handled? What happens if that party disappears? Design the custody chain, and the mechanisms mostly select themselves.

The thing that makes it hard: failures are quiet

Storage failures are loud: the disk is full, the query errors, the table is missing. Movement failures are quiet.

• Duplicates look like data.
• Reordering looks like data.
• A silently dropped route looks like nothing at all.
• A partition that stalled looks like a metric that stopped moving, which looks like a metric that had nothing to report.

So the monitoring requirement is different in kind. In storage you monitor for errors. In movement you monitor for absence and asymmetry: counts that do not reconcile from end to end, a maximum that diverges from the mean, a lag that has stopped changing, an age that only grows. If your monitoring can only tell you that something went wrong, it is the wrong monitoring for this domain. It must tell you that something did not happen.

Example. Orders counts 100,000 OrderPlaced events in an hour. The warehouse counts 99,412. No error was logged anywhere. Only a reconciliation of the two counts shows the gap of 588 events, and only a per-partition view shows that all of them belong to one partition.

Why movement constrains everything else

Movement decisions are made early, encoded in infrastructure, and depended on by everything downstream. So they set the ceiling for domains you have not designed yet:

• Your storage layer can be only as correct as what arrives.
• Your computation layer can be only as parallel as your routing allows.
• Your semantics (can we reprocess? can we fix a bug retroactively? can we add a consumer without a project?) were decided by the transport choice, often by whoever picked the default years ago.

There is a practical asymmetry. Most of a data platform can be rewritten step by step. A partition key, a retention window or an ordering guarantee cannot. These spread into every consumer's assumptions, and undoing them is a migration with a compatibility window, not a refactor. Movement is where you should spend design time out of proportion to how much of the codebase it occupies.

The shape of the whole domain: relocated coupling

Every mechanism here was buying decoupling. But decoupling is not free. It is relocated coupling. You never remove the dependency between producer and consumer. You move it somewhere you would rather have it. (This is the same law as "costs are conserved and relocated": a cost is never removed, only moved to another place, time or person.)

• Asynchronous delivery does not remove the coupling. It converts a synchronous dependency into a shared retention window and a shared dedup contract.
• A broker does not remove the dependency. It makes the broker the dependency.
• Partitioning does not remove coordination. It relocates it into the choice of key.

So the honest summary of this mechanism is: a catalog of places to put coupling, and the relative cost of each location. The skill is not knowing the mechanisms. The skill is knowing where a given system can afford to be coupled, and routing the unavoidable dependency there.

How the domain fails as a whole

Two patterns, both structural rather than tied to one mechanism.

Coupling reappears where nobody is looking

• Retention becomes a cross-team contract with no interface.
• A replication slot ties the uptime of analytics to the survival of the production database.
• A shared partition count silently caps the parallelism of three teams.

These dependencies exist at runtime and appear in no diagram, no schema and no code review. Movement architecture collects invisible edges, and incidents are usually the first time anyone draws them.

Local optimizations compose into global pathology. Each layer independently added buffering, retries or timeouts that were correct on their own. Together they produce amplified retries, multiplied latency, or a metastable state that survives the removal of its trigger. This domain rewards thinking in whole paths and punishes thinking in components. That is unusual, because most engineering rewards the opposite.

The single organizing idea

If you keep only one thing from this mechanism:

This is not a metaphor. Every choice here was a specific answer to that question:

Systems that feel solid are not the ones that lose nothing. That system does not exist. They are the ones where every loss was chosen, bounded and visible. Systems that feel fragile are the ones where the losses were inherited from defaults and discovered during an incident.

That is the real competence this domain tests: not whether you can name the mechanisms, but whether you can look at a running pipeline and state, precisely, what it is currently willing to lose, and whether anyone ever decided that.