Data Movement
How do records cross between two systems that have independent rates, independent availability and independent lifetimes, and what are you willing to lose when the assumption underneath the crossing breaks?
Who this mechanism is for: a developer who writes services, calls APIs and knows basic SQL, but has never designed a message broker, run Kafka, or reasoned about delivery guarantees. You do not need any distributed-systems background. Every technical term is explained the first time it appears, and again in the glossary at the end. When an idea from the physics of a single machine is needed (a disk seeks, a cache line is 64 bytes, a queue that is 80% busy is already slow), we explain it again in a few sentences where it is used.
The story we will follow
Cartwheel is a small online grocery marketplace. Customers place orders, and independent sellers fulfil them. Ravi is the backend engineer who owns the Orders service. Lena is the data engineer who owns the analytics pipeline and the event platform. Omar, the CTO, asks the blunt questions. Sana is often the engineer on call when things break.
When the company started, the Orders service simply called every other service directly: Payments, Inventory, Email, Fraud scoring. Now Cartwheel is growing. Over the next weeks Ravi and Lena will move it to an event-driven platform, where services pass records to each other through brokers and logs instead of calling each other. Along the way they will meet six problems, one at the start of each article below: a failed checkout, a double confirmation email, a crashed consumer, a stalled partition, a network bill that is too high, and a replay that doubles last month's revenue.
Each problem looks like a bug. Each one is actually a consequence of something deeper about how two separate computers talk to each other. At the end of each article we go back to Ravi and Lena and solve the problem with the ideas of that part. (All story numbers are illustrations, not measurements.)
This is the main idea of this whole mechanism:
How this mechanism is organized
- 1
From one machine to two
Why is data movement a different kind of problem from anything inside one machine?
CaseEmail is down, so every checkout fails
- 2
Transport patterns
How do records cross: queue, log, publish/subscribe, RPC, push, pull, change feeds, file drops, and what does each promise?
CaseRecords gone, a stalled partition, two emails
- 3
Flow control
What happens when records arrive faster than they can be handled?
CaseBlack Friday: the consumer killed by its own buffer
- 4
Routing and distribution
Which of N destinations gets a record, and what happens to order and balance when you split a stream?
CaseOne partition hours behind, and time running backwards
- 5
Network-level mechanisms
Which low-level choices (framing, compression, pooling, heartbeats, TLS) change behavior, and which are only constant factors?
CaseA high bill, morning spikes, connections that hang
- 6
Delivery across a pipeline
Where do loss, duplicates and reordering come from across a whole pipeline, how does a replay fix the past, and how do you find out what you really have?
CaseA replay doubled last month's revenue
- 7
The whole domain
How do the four groups of decisions compose, and what kind of domain is data movement as a whole?
After that: a summary, hands-on labs, interview questions and a glossary.
Note on the structure: the four groups (transport, routing, flow control, network mechanisms) are not four equal topics. They are a stack. A real pipeline picks one choice from each group at the same time. From one machine to two explains this. Delivery semantics is not a fifth topic beside them; it is where the correctness of the whole stack lives, so it appears in Transport patterns and gets Delivery across a pipeline. Making a repeat harmless is Fault tolerance's subject.
What do I do, and what does each option cost?
Choose a transport
Records must move from a producer to consumers: a synchronous call, a queue, publish/subscribe or a log? Three questions pick the transport; five settings then make it safe.
ChooseHandle overload
Work arrives faster than it can be served: break the circuit to a failing dependency, buffer a burst, slow the source, refuse at the door, or shed? Four questions pick the response; two checks keep it from spreading.
ChooseChoose a partitioning
Records are split across N partitions or shards: round-robin, hash or range, by which key, with how many partitions, and what happens to skew, order and failures?
ChooseMake effects repeatable
Messages can arrive more than once and consumers can crash mid-way: list every effect, make each one safe to repeat, commit progress after the work, and prove it by crashing the consumer.
Design
Can I see it myself?
Head-of-line blocking: log versus queue
Simulate 12 partitions with 100 records/s each and one poison record.
SimulatePythonPer-record calls versus batches: find the natural batch size
Send 100,000 records to a local HTTP server in batches of 1 to 10,000 and fit the fixed cost per call.
MeasurePython with requestsAck before or after processing: loss versus duplicates
Kill a consumer at random moments: saving the offset first loses records, processing first duplicates them.
Break itPythonBufferbloat: a bigger buffer buys latency, not throughput
A producer 20% faster than its consumer, with queues from 100 to 100,000: throughput stays put, latency grows with the queue.
SimulatePythonWhat an overflowing buffer loses
A keyed stream overloads a bounded buffer under drop-newest, drop-oldest and drop-oldest per key: count the records dropped and the entities left with a wrong value.
SimulatePythonBatch size: find the knee
Insert 200,000 rows into SQLite with one commit per batch, then bound trickle-traffic latency with a size-or-age rule.
MeasurePythonToken bucket versus fixed window
Two bursts straddling a one-second boundary against a fixed window, two token buckets and a sliding window.
SimulatePythonMeasure key skew and the effect of salting
Partition 1,000,000 events by seller, by order and by salted seller, when one seller owns 40% of them.
SimulatePythonHow many keys move when N changes? Modulo vs. ring vs. rendezvous
Add an eleventh node: how many of 100,000 keys move under modulo hashing, a ring with 1 or 128 virtual nodes, and rendezvous hashing.
SimulatePythonBreak per-key order in a real Kafka topic, then repair it
Per-key order in Kafka, broken three ways (a worker pool, lost acks without idempotence, growing the topic) and repaired twice.
Real system · Break itDocker, PythonA DLQ that does not become an outage amplifier
Write a small consumer with a retry-then-DLQ policy.
SimulatePythonChoose N by measurement
Measure one consumer, derive the partition count, then compare N and 2N partitions for recovery time and files per hour.
Real system · MeasureDocker, PythonWhat a rebalance repeats
Redeploy one consumer of a 12-member group on a 24-partition topic, many times: count what eager, cooperative and static membership process twice.
SimulatePythonThe length-prefix attack and the maximum frame size
A server that trusts a 4-byte length header, a header claiming 4 GB, and one wrong length in the middle of a stream.
Break itPythonBatch size is a compression parameter
Compress 100,000 small JSON events in batches of 1 to 10,000, then with a trained dictionary.
MeasurePython with zstandardCompress or not: the rate comparison
compress + transfer + decompress against sending raw, for five codecs and three link speeds.
MeasurePython with lz4, zstandardStale pooled connections: find out what your pool does
Exhaust a pool of 2, kill a proxy silently under a pooled connection, and read your HTTP library's defaults.
Break itPython with urllib3Nagle plus delayed ACK: reproduce the 40 ms cliff
Two small writes and a short reply: Nagle on the sender and delayed ACK on the receiver add about 40 ms per call.
MeasurePython (Linux shows the effect)Dedup window versus replay
A 60-second dedup window against a replay five minutes later, then idempotency by replacing the partition.
SimulatePythonReconciliation count and duplicate-rate metric
Count records per hour at source and sink, then plant a bug that swallows 1% of writes: errors stay at zero, the count gap does not.
SimulatePython
Can I reason about it under questioning?
Interview questions
Short questions on moving data between systems, such as queues and logs, flow control, routing and delivery guarantees, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on moving data between systems, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Ordering vs. parallelism
Global order costs serialization; parallelism costs order.
Exactness vs. approximation
For example: sketches, sampling, ANN.
Push vs. pull
Who initiates the work.
Fail-fast vs. fail-degraded
Reject or continue when something is wrong.
Latency vs. throughput
For example: batching, group commit, micro-batch.
Show all 12 trade-offsShow fewer
Coupling vs. coordination cost
For example: 2PC vs. sagas, schema contracts vs. schema-on-read.
Durability vs. acknowledgment latency
How much must be confirmed before you say "yes".
Early binding vs. late binding
When structure is committed; determines where the failure surfaces, at write time or read time.
Read-optimized vs. write-optimized
For example: B-tree vs. LSM, merge-on-read vs. copy-on-write.
Space vs. time
For example: compression, indexing, materialization, caching.
Generality vs. specialization
For example: general engines vs. purpose-built structures.
Mutability vs. immutability
In-place update vs. append-only + GC; buys reproducibility and cheap concurrency, pays in space and janitorial work.
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Variance costs more than volume
Variance costs more than volume.
Costs are conserved
Costs are conserved, never removed.
Fixed cost per operation
Fixed cost per operation implies a natural batch size.
Guarantees have scope
Atomicity is bounded by the shared commit substrate.
Coordination cannot be made cheaper
Coordination is the only thing you cannot make cheaper.
Show all 17 principlesShow fewer
Local optimizations compose badly
Identical mechanisms at adjacent layers do not add up. They interfere.
Granularity mismatch
Granularity mismatch creates amplification.
The algebra decides
What can be skipped, pushed down, parallelized, merged or precomputed is determined by the algebra of the function, not the layout of the data.
The end-to-end argument
Guarantees compose to the weakest link, and only endpoint guarantees are real.
Little's Law
Concurrency = Throughput × Latency.
Queues explode near full load
Response time explodes as utilization nears 100%.
Independence is assumed
Independence is assumed everywhere and is true almost nowhere.
Absence is not evidence
Absence of an event is not absence of a failure.
Sound approximations
When exactness is too expensive, use a sound approximation.
Hidden state breaks replay
A replayable pipeline is a pure function of its retained inputs.
Power laws and skew
Undeclared dependencies
An optimization that succeeds becomes an undeclared dependency, with leverage 1/(1 − h).