Eight transports on three switches
A transport is the thing that sits between a producer and a consumer that have independent rates, availability and lifetimes. Every pattern below is a specific purchase of decoupling along those three dimensions, plus a specific answer to four consequential questions:
- Who holds the record between send and process? (the sender, a broker, a network buffer, or nobody)
- Who holds the progress state? ("how far have we got?": the broker per message, the consumer per stream, or nobody)
- What survives a crash on either side?
- What order is preserved, and over what scope?
Answer these four for any transport and you have characterized it. Everything else (protocol, vendor, API shape) is detail.
The lattice: eight patterns on the three switches
| Pattern | Rate decoupled | Availability decoupled | Identity decoupled | Progress state lives | Replay |
|---|---|---|---|---|---|
| Sync RPC / request-response | no | no | no | nowhere | no |
| Streaming RPC | partial (credits) | no | no | the connection | no |
| Multicast (UDP) | no | no | yes | nowhere | no (only gap filling) |
| In-memory queue | yes (bounded) | no | partial | broker (volatile) | no |
| Durable queue | yes | yes | partial | broker, per message | no |
| Pub/sub over durable queues | yes | yes | yes | broker, per subscriber | no |
| Append-only log | yes | yes | yes | consumer (an offset) | yes |
| Bulk file transfer | yes (unbounded) | yes | yes | filesystem / file naming | yes |
Replay means reading the same records again after they were already processed once.
Two things fall straight out of this table.
First: the durability column and the replay column are not the same thing. A durable queue survives a broker restart, but it cannot give you the record back once it is acknowledged. Durability protects against a crash. Retention (keeping records for a period) protects against being wrong. Most teams buy the first and discover a year later that they needed the second. This is Lena's first problem exactly.
Second: the "identity decoupled" column is what makes a system extensible. You can add a consumer without touching the producer. This is what people actually mean by "loosely coupled", and it is cheaper than they think: it costs an addressing indirection (a topic name) and nothing else. The expensive decoupling is availability, because that one requires storage.
Primitives, composites and shapes
A common list of "transport patterns" mixes three levels:
- Primitives (irreducible; each is a real decision): the append-only log, the queue, request/response, push versus pull.
- Composites (a primitive plus a policy; learn them as derivations, not as equals): publish/subscribe, point-to-point, streaming replication, change feeds, long polling, RPC variants, bulk transfer.
- Shapes (descriptions of how many senders and receivers, which apply to any of the above): fan-in, fan-out, broadcast/multicast.
So there are four things to understand deeply, and then a set of derivations. We follow that order.