Streaming
What does it cost to remember, when the input never ends?
Who this mechanism is for: a developer who writes code, calls APIs and knows basic SQL, and has maybe used a message queue, but has never built or operated a stream-processing job. You do not need any database or distributed-systems background. Every technical term is explained the first time it appears, and again in the glossary at the end. Where an idea comes from a topic that other books treat separately (queues, transactions, log compaction, queueing mathematics), we explain it in a few sentences right here.
The story we will follow
Ana Costa is a backend engineer at Ridelo, a food-delivery company. Every order, cancellation, payment and courier GPS ping becomes an event (a small record that says "this happened") and is appended to a shared log. Ridelo runs a stream-processing job on that log: a program that never finishes, reads events as they arrive, and keeps results fresh. The job computes fraud scores per user, joins orders with courier positions, enriches orders with restaurant data, feeds a "busiest restaurants in the last 24 hours" dashboard, and writes charges into the billing database.
Her teammate Tomás is the on-call engineer. Ana is new to streaming. In nine parts she will meet nine problems. Each article starts with a problem that looks like a bug: a crash after three quiet weeks, a cluster that will not scale, a number that is slightly too small, a bill that is charged twice. Each article ends with the solution, using the mechanisms we just learned. The solutions are not clever patches. They follow from one fact, and that fact is the main idea of this whole mechanism:
(Batch processing means a job that reads a fixed amount of input, produces an output and ends. State means the data a program must remember between events.)
How this mechanism is organized
- 1
The job that never ends
What changes when there is no end of input, and which five questions cover the whole topic?
CaseThree flat weeks, then out of memory
- 2
State
How big is the state, who owns it, where does it live, and how does it expire?
CaseOne hot restaurant, and the job keeps freezing
- 3
Time-shaped state
How do we keep state bounded when results are defined over time?
CaseA 24-hour sliding window pins the CPU
- 4
Joins and enrichment
How do we combine two streams, or a stream and a table, without unbounded memory or wrong answers?
Case2% of deliveries missing from the join
- 5
Changelogs and queryable state
What does a streaming job actually emit, and can we read its state directly?
CaseA maximum that never drops
- 6
Checkpoints and exactly-once
How do we recover without re-running from the beginning, and what does "exactly once" really mean?
CaseCheckpoints from 20 seconds to 9 minutes
- 7
Rescaling and upgrades
What happens to state when we change parallelism or code?
CaseA 14-minute rescale, and state that will not load
- 8
Backpressure and batching
What happens when one part is slower than the rest, and is micro-batching different from record-at-a-time?
CaseFour operators red, and one culprit
- 9
What streaming forbids
What does streaming forbid, what do the escapes cost, and what can we never verify?
Case1.5% short, and no alert fired
After that: a summary, labs, interview questions and a glossary.
One request: read the "Rule" boxes at the end of many sections. Each one is the section in a single sentence.
What do I do, and what does each option cost?
Can I see it myself?
Key-bounded versus time-bounded state
A per-user counter and a 600-second join buffer over the same stream at two rates, then the buffer without eviction.
SimulatePythonThe indivisible key
Hash partitioning over 8, 32 and 128 instances with one key carrying 30% of traffic; then salting it into 8 sub-keys.
SimulatePython with NumPySliding window with panes, and floating-point drift
A 24-hour sliding count refreshed every minute, kept per window and with 1-minute panes; then a float sliding sum by add-new, subtract-old.
MeasurePythonAligned versus unaligned checkpoints under backpressure (optional, needs a local stream engine)
A model of barrier checkpoints behind a bounded buffer and a slowing sink: duration and size, aligned against unaligned.
SimulatePythonReplay determinism
An exactly-once loop with checkpoints, crashed at random and restarted 50 times: a pure computation and one that reads random.random(); then an upsert sink.
Break itPython
Can I reason about it under questioning?
Interview questions
Short questions on stream processing, such as state and its size, windows and watermarks, stream joins, changelogs, checkpoints and exactly-once, rescaling and backpressure, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on stream processing, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Early binding vs. late binding
When structure is committed; determines where the failure surfaces, at write time or read time.
Coupling vs. coordination cost
For example: 2PC vs. sagas, schema contracts vs. schema-on-read.
Isolation vs. utilization
For example: multi-tenancy, resource pools, spot capacity.
Push vs. pull
Who initiates the work.
Durability vs. acknowledgment latency
How much must be confirmed before you say "yes".
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Hidden state breaks replay
A replayable pipeline is a pure function of its retained inputs.
Retention is paid by the live path
Costs are conserved
Costs are conserved, never removed.
Granularity mismatch
Granularity mismatch creates amplification.
Fixed capacity meets growth
A fixed-capacity structure meeting growing cardinality breaks the design.
Show all 11 principlesShow fewer
Variance costs more than volume
Variance costs more than volume.
Immutable storage grows indirection
An immutable substrate grows an indirection layer plus a collector.
The algebra decides
What can be skipped, pushed down, parallelized, merged or precomputed is determined by the algebra of the function, not the layout of the data.
Independence is assumed
Independence is assumed everywhere and is true almost nowhere.
Absence is not evidence
Absence of an event is not absence of a failure.
Masking moves failure
Masking a failure moves it from a channel with a schema to a channel without one.