Latency vs. throughput
Where you meet it: batching, group commit, micro-batch.
What it is
When the fixed cost is large, the sensible move is to batch: pay the setup once and spread it over more bytes. Think of an elevator: it is wasteful to go up with one parcel, so you wait until you have a full trolley. But batching means the first item waits for the last item. This is the latency versus throughput trade-off. It is not a separate law; it is a consequence of non-zero fixed cost. If setup were free, batching would give nothing.
From Physical Substrate & Performance Laws › The dial and the fixed cost
Latency to the first row rises by one batch.
From Execution Engine Runtime › Operators and handoffs
Sneakernet (physical disk shipping, such as AWS Snowball) belongs on this list, seriously. Compare bandwidth × time with a truck. Worked example: moving 1 PB (1,000 TB) over a sustained 1 Gb/s link takes 10¹⁵ B × 8 / 10⁹ b/s = 8 × 10⁶ s ≈ 93 days. Past a few hundred terabytes, the truck wins on throughput by about an order of magnitude, while its latency is measured in days. It is the purest example of the latency-versus-throughput trade-off: the two ends are so far apart that they use different vehicles.
From Data Movement › CDC, replication, files
Where it appears
In 10 mechanisms. Each says what the mechanism is about, then how this trade-off shows up there, in its own words.
- Physical Substrate & Performance Laws
Why do trade-offs exist at all?
This is the latency versus throughput trade-off.
Read it inStorage media: what each one is made of and what it forces
- Data Movement
How do records cross between two systems that have independent rates, independent availability and independent lifetimes, and what are you willing to lose when the assumption underneath the crossing breaks?
Latency versus throughput is the master trade-off. It appears at every layer.
Read it inWhat changed: from one machine to twoChange feeds, replication, bulk transfer and the shapes
- Persistence & Durability
When a system says "yes, your data is safe", what exactly has it promised, and what did it have to do to keep that promise?
You trade a fixed, bounded latency penalty for an unbounded gain in throughput.
Read it inGroup commit
- Encoding, Serialization & Compression
What do we turn our values into when we turn them into bytes, and what does that choice cost?
For a point-lookup workload this difference is the latency budget.
- Memory & Resource Management
Who is allowed to hold how much memory, for how long, and what happens to the work when the answer is less than it asked for?
Everything between the two is a system that will page you at night.
Read it inGC tuning versus manual management
- Caching
What happens when we deliberately keep a second copy of data that no protocol forces to be correct?
The design guidance is the opposite of what teams usually do: cache in front of the slowest thing, not the most-requested thing.
Read it inMiss penalty and the percentile you promisedThe tier-insertion rule
- Distributed Systems Coordination
Who is allowed to decide, and how does everyone else learn the decision without asking?
Hedging converts a latency distribution into the minimum of k draws, at a cost of k× work on the hedged fraction only.
Read it inHedging
- Scheduling & Workload Management
Which work runs now, which work waits, and who pays for the waiting?
So state it flatly, because it is sold wrongly: reactive autoscaling handles trends, never spikes.
Read it inDelay schedulingAutoscaling
- Observability & Operations
What is the smallest description of a running system from which we can find a fault, fast enough to matter, at a cost we can still pay while the system is failing?
For data systems, the dominant time is off-CPU time: the program is waiting for I/O, a lock, or a shuffle partner.
Read it inProfiling
- Measurement & Experimental Method
What would have to be true for this number to mean what I think it means?
Capacity is the rate at which the SLO still holds, not the rate at which the machine is busiest.
Read it inThroughput at an SLO, and goodput
Related
Made on the same pages most often.