What changed: the epistemic limit
What can a system no longer know once its data has to cross between two machines?
Before this mechanism, the limits we cared about belonged to a single machine. A disk seeks. A cache line is 64 bytes. A queue at 80% utilization has already used its latency budget. Everything was measurable and repeatable, and, importantly, knowable from where you stood. You could attach a probe. Physics and arithmetic do not lie to you.
Data movement adds one new thing, and it changes the character of every problem:
That is the whole reframe. It is not "things are far apart now". Distance is only latency, and latency was already a known cost. The new limit is informational:
- You can send a message, but you cannot know it arrived.
- You can wait for a confirmation, but the confirmation can be lost.
- You can wait longer, but you cannot tell slow from dead. Not with a better timeout, not with a better network, not ever. This is a proven impossibility, not an engineering gap.
The limits of a single machine were physical. The limit here is epistemic (about what you can know). You are no longer limited by what hardware can do. You are limited by what you can find out.
This is exactly what happened to Ravi. The Payments call timed out. Orders could not know whether Payments (a) never got the request, (b) got it and is still working, (c) finished it and the reply was lost, or (d) crashed in the middle. From where Orders stands, all four look the same: silence.
What stops mattering
Two things from single-machine thinking fade. Noticing this is how you know you have changed layers.
Precision stops mattering. Inside one machine you care whether an operation takes 100 ns or 300 ns, because you do billions of them and a factor of three is your whole margin. Here, a network round trip is about 10,000 times a memory access. Nanosecond-level reasoning is noise. You reason in orders of magnitude and in counts of round trips: one hop or three, one ack or a quorum (a majority of replicas). If you are tuning a serialization routine while paying an extra network round trip per record, you are working on the wrong problem by four orders of magnitude.
Determinism stops mattering, because it stops existing. On one machine you are rewarded for removing variance: predictable access patterns, branch-free code, aligned I/O. Here variance cannot be removed:
- Packets are dropped, retransmitted and reordered.
- Nodes pause for garbage collection (GC).
- Leaders change.
- A message that took 2 ms a thousand times takes 30 seconds once, for reasons nobody will ever explain.
You stop trying to make behavior deterministic. You start trying to make outcomes correct despite non-determinism. This shift, from prevention to tolerance, is the intellectual move of this whole mechanism.
What carries forward in a new form
The laws of a single machine do not stop. They reappear at a bigger scale, where the numbers are larger and the consequences are about meaning, not only speed. Recognizing them is most of the compression you can get. You have probably met most of this mechanism's mechanisms already, inside one machine, where the failures were slow instead of wrong.
Granularity mismatch creates amplification. Granularity is the smallest piece a mechanism handles. If the unit you have does not match the unit the substrate wants, you pay a multiplier (an amplification). Inside one machine: writing 4 KB into a flash block that can only be erased as 256 KB. Here: a 200-byte record sent in a 1,500-byte network packet, or one row written into a 128 MB file.
A fixed cost per operation produces a natural batch size, and the fixed cost moves upward. Every operation has a setup cost F paid no matter how big the operation is. If you handle B records per operation, the per-record cost is F ÷ B. Inside one machine, F was a disk seek, then a system call. Here F is the network round trip, the broker request, the API call. The formula does not change. Only the dominant F changes. Batching is the same mechanism, but it now has a semantic cost too: a batch fails as a unit.
Little's Law connects latency and throughput. It told you how many requests to keep in flight on a fast SSD. Here it sizes your connection pool, your consumer concurrency, your credit window (how much a receiver allows the sender to send) and your in-flight request limit. Same algebra, different units.
Variance costs more than volume. A slow disk gave a slow tail on one machine. Here you get tail amplification: a request that fans out to ten nodes waits for the slowest, so your median can become their p90 (the slowest 10%). Distribution does not average the variance away. It compounds it.
Costs are conserved: they only move. This is the most important one. Every mechanism here moves a cost somewhere else instead of removing it: from the writer to the reader, from time to space, from your team to the on-call team. When you hear "free win", ask which party absorbed the cost.
The new laws of this mechanism
Three laws could not be taught on a single machine, because they need a second machine to exist.
1. Independence is assumed everywhere and is true almost nowhere. Distribution creates the appearance of independence while leaving the reality of shared fate. Examples:
- three replicas that live on one rack;
- two services that sit behind one shared dependency;
- ten thousand clients whose retry timers are synchronized by the same outage.
The system was designed for independent failures and will meet correlated ones. (Correlated disk failures and shared power were an early hint of this, and here it becomes the dominant failure mode.)
2. Coordination is the only thing you cannot make cheaper. You can add bandwidth, add machines, add cache. You cannot make agreement between two parties cost less than one round trip, and you cannot make it happen at all across a network partition (a break in communication between groups of machines). This is why every architecture in this field is, underneath, an attempt to avoid needing to agree: partition by key so that nobody must coordinate, use immutable logs so that nobody must lock, choose operations whose order does not matter. Distributed-systems design is largely the art of not needing consensus.
3. Every guarantee has a scope, and the scope is the whole claim. "Ordered" means nothing. "Ordered per key, per producer session, within one partition, when there are no retries" is a claim you can check. On one machine a guarantee was usually total: the disk either wrote the block or it did not. Here, guarantees without a stated scope are the main source of production surprises, and they usually come from a vendor's marketing page and not from anyone's decision.
The governing question
Inside one machine the question was: how fast can this substrate go, and what does it charge? For data movement it is:
This framing is generative: it produces the mechanisms instead of cataloguing them.
- Rates differ → you need buffering, batching, backpressure (Flow control).
- Availability differs → you need persistence, retries, replay (Transport patterns and Delivery across a pipeline).
- Lifetimes differ → you need addressing indirection: topics, brokers (Transport patterns and Routing and distribution).
Four sections of material fall out of three sources of independence. And the framing makes the trade explicit. There is no configuration in which nothing is lost. There are only configurations where the losses were chosen.
Why this is the layer where mistakes are expensive
A single machine tells you what is possible. Data movement is the first place where you make promises: that a record will arrive, that order will hold, that nothing will be duplicated. Everything built on top depends on those promises being true, and none of it can verify them independently.
That gives data movement the worst error propagation of any layer. A mistake on one machine is slow. A mistake here is wrong, silently, downstream, in someone else's system, weeks later. Duplicates look like data. Reordering looks like data. A dropped route looks like nothing. This is why the layer deserves disproportionate design attention: the mechanisms are simple, the failures are not, and the decisions are the hardest in the platform to reverse.