What changed: from one machine to two
Why is data movement a different kind of problem from anything inside one machine?
What "data movement" actually is
Strip away the vocabulary. Every queue, topic, API call and file drop here is an answer to one problem:
Look at the three words:
- Independent rates. The producer may send 2,000 records per second while the consumer handles 800.
- Independent availability. The consumer may be down for a deploy, a crash or a bad day, while the producer keeps running.
- Independent lifetimes (identity). The producer should not need to know who the consumers are, how many there are, or whether new ones will appear next year.
Every transport pattern is a choice about how much to decouple along these three dimensions. Each kind of decoupling has a price:
| Decoupling | Bought by | Paid for in |
|---|---|---|
| Rate (producer faster than consumer) | buffering | memory or disk, latency, staleness |
| Availability (consumer is down) | persistence (storing records until the consumer returns) | durability cost, ack latency, cleanup (garbage collection) |
| Identity / lifetime (producer does not know who consumes) | an addressing indirection (a topic, a broker) | coordination, ordering problems, more to operate |
Now the two extremes:
- RPC (a plain call to another service and wait for the reply) decouples none of the three. That is why it is cheap and simple, and why it fails hard: Cartwheel's Tuesday outage is exactly this.
- A durable log (a stored, ordered sequence of records; defined in Transport patterns) decouples all three. That is why it is expensive and why it has become the default answer to almost everything.
Everything between those two poles is a partial purchase.
A useful habit: before you reason about any transport, place it on this three-switch table: rate decoupled yes or no, availability decoupled yes or no, identity decoupled yes or no. Treat it as three on/off switches (a "3-bit lattice", meaning eight possible combinations). Transport patterns does this for eight real patterns.
The four groups are a stack, not four peers
The subject "data movement" is often listed as four separate topics. They are not siblings. They compose. A real pipeline picks one choice from each layer of this stack at the same time:
- Transport pattern: the semantics of the handoff. Does the consumer get a copy, or does it claim the record? Is the transport itself the system of record? (The key distinction here is log versus queue, not publish/subscribe versus point-to-point. Transport patterns explains why.)
- Routing: which of N destinations receives the record, and what survives of the order once you have split the stream.
- Flow control: what happens when the rate assumption breaks. This mechanism exists only because the decoupling of the transport layer is finite (a buffer is never infinite).
- Network mechanisms: the substrate. Mostly transparent, mostly not about meaning.
A good test of your understanding: take a product and place it in a cell of each layer. Kafka is not "a transport pattern". It is an append-only log (transport) plus key-based partitioning (routing) plus pull delivery (transport) plus consumer-paced flow control with no credits (flow control) plus length-prefixed framing over TCP (network). Being able to take a product apart like this is the real skill.
Which trade-offs bind here
Several trade-offs run through this whole mechanism. It helps to name them now, in plain words:
- Latency versus throughput is the master trade-off. It appears at every layer.
- Push versus pull (who decides when data moves) and ordering versus parallelism are the two that make transport decisions hard to reverse. Once you have partitioned for parallelism, you cannot get global order back without sending everything through one point again.
- Durability versus acknowledgement latency is the most under-appreciated one. "Sent", "buffered", and "fsynced on N replicas" are three very different promises. The choice is the single biggest determinant of both latency and what you can promise downstream.
- Fail-fast versus fail-degraded is the whole point of flow control. Backpressure, load shedding, dead-letter queues and circuit breaking are four different answers to the same overload event.
- Coupling (how much two systems must know about each other). Request/response and publish/subscribe differ mainly in addressing, which is a form of coupling. The expensive coordination questions (two-phase commit, data contracts) live elsewhere.
- Early versus late binding: when do you find out that a record is bad? Protocol framing and the wire schema decide whether a bad record fails when it is serialized (early) or when someone parses it (late). This belongs to the network mechanisms.