Choose a transport
A producer creates records, say orders being placed, and other systems need them. You can call the consumer directly and wait for an answer, put each record on a queue for one worker, broadcast it to every interested consumer, or append it to a log that consumers read at their own pace. Which transport fits, and what must you settle once you have picked it?
You will end with a transport, plus its loss window, ordering key, batching, full-buffer policy and deduplication point.
Decide
Does the sender need the result before it can continue?
Checkout needs the payment authorization now; it does not need the warehouse's acknowledgement.
Why
A synchronous call is the only transport that gives the sender an answer, and it couples the two: when the callee is slow or down, so is the caller.
Will anyone ever need to read these records again after they are processed?
A new consumer, a bug fix, a backfill, an audit, debugging. Be honest: 'no' is almost always wrong for facts (an order was placed) and almost always right for tasks (resize this photo).
Why
A durable queue survives a crash but deletes a record once it is acknowledged. A log keeps records for a retention period, and each consumer tracks its own position, so any consumer can go back. Durability protects against a crash; retention protects against being wrong.
Should each record go to one worker, or to every interested consumer?
Why
A queue hands each record to one of many workers, so work spreads across them. Publish/subscribe gives every subscriber its own copy, so new consumers can be added without touching the producer.
Where you can land
Synchronous call (RPC)
Use it when the sender needs the answer to proceed.
What it costs: availability coupling: the caller fails or slows with the callee. Keep such calls on the request path; batch work over millions of records should not make one call per record.
Log
Use it when records are facts, more than one consumer will want them, order per entity matters, or you will reprocess.
The default in data engineering, because records are facts and consumers multiply.
What it costs: retention storage, and head-of-line blocking: records in one partition are consumed in order, so one slow record holds up the rest of its partition.
Queue (point-to-point)
Use it when each record is an independent task, its cost varies, and history has no value.
Workers take tasks as they free up, and failed tasks are retried individually.
What it costs: no replay. Running tasks on a partitioned log instead gives idle consumers next to a stalled hot partition, and no per-task retry.
Publish/subscribe
Use it when every interested consumer needs each record, but nobody needs to re-read old ones.
What it costs: a subscription per consumer, managed by the broker, and still no replay once delivered.
Whichever you land on, settle these
The loss window
How many seconds of acknowledged records may be lost in a failure? That sets the acknowledgement level, replication mode and persistence.
What must stay in order, and by which key
That sets the partitioning, and therefore caps how many consumers can work in parallel.
Latency budget against record rate
That sets push or pull, and the batch size.
What happens when consumers cannot keep up
Block, buffer, shed or drop. See Handle overload.
Where duplicates are removed
Name the key and the store, in one sentence. If you cannot: you have an at-least-once pipeline with no deduplication, which is a pipeline that produces duplicates. This is the question most often skipped, and it produces the bugs that take three days to find.