Change feeds, replication, bulk transfer and the shapes

Which composite transports carry records between whole systems, and what does each fan-out or fan-in shape cost?

The derived patterns

These are composites: a primitive plus a policy.

CDC (change data capture) means taking a database's own record of changes and turning it into a transport, so that other systems can follow the database. Cartwheel's Orders service writes to PostgreSQL, and Lena wants those changes in the warehouse without asking developers to publish events by hand. There are two implementations, and the choice is again push versus pull.

Query-based (pull / diff polling) repeatedly selects the rows changed since a high-water mark. It needs no special privileges and works on any database, but it misses deletes and intermediate states, and it depends on a trustworthy, always-increasing column (what polling can never see).
Log-based (tailing the WAL / binlog / oplog) reads the log the database already writes, so it is complete and in commit order, at the price of privileges and coupling to the engine's log format. It also creates the most severe coupling in this whole space: a stalled consumer stops the source from recycling its log (Log-based CDC; Backpressure follows it upstream).

Starting a feed on a table that already exists needs a snapshot that meets the stream with no gap and no overlap: Bootstrap and represent.

Streaming replication and log shipping

This composite is an append-only log plus push, with a receiver that applies the records (rebuilds state) instead of just consuming them. The interesting decision is when the primary tells the client "committed": after its local commit (async), after a replica has received the bytes (semi-sync), or after N replicas have them durably (sync). The loss window of each, and the trap that "semi-synchronous" usually means received and not fsynced, are in Durability across machines.

A second decision: physical versus logical.

Physical (byte or page level): fast, exact, no interpretation. It needs identical versions and layouts. The replica is a byte-identical clone. You cannot filter, transform or replicate across engine versions.
Logical (row-change events): it crosses versions and engines, allows filtering and transformation, and enables selective replication and zero-downtime upgrades. It costs CPU to decode and re-apply, and it cannot replicate everything (sequences, DDL and large objects have caveats in most implementations).

This is the generality-versus-specialization trade-off (a general format works everywhere but costs more; a specialized one is fast but narrow), and again early versus late binding.

Bulk transfer

Bulk transfer (file drops: write files to a shared location, and the consumer picks them up) gives maximum decoupling on every dimension. The buffer is a filesystem, effectively unbounded. The availability coupling is zero. The producer names a location, not a consumer.

Some under-rated strengths:

The file is a natural transaction unit. Write to a temporary path, then atomically rename it into place. The consumer sees all or nothing, with no protocol needed.
The filename is a natural idempotency key. Reprocessing becomes "have I seen this name?". This is a dedup mechanism you get for free, and most streaming systems need extra infrastructure for it.
Replay is trivial. The data is just sitting there.

Costs:

The latency floor is the file interval. It cannot be negotiated: if you drop a file every 5 minutes, data is at least up to 5 minutes old.
The completeness problem. Never decide that a file is finished by polling its size. You will read partial files, and the corruption will be intermittent and awful to debug. Use an atomic rename into the watched location, or a _SUCCESS marker file written after the data, or a manifest (a list of files that belong together).
Error granularity is the file. One malformed record forces a whole-file decision: reject everything, or accept the file and quarantine the bad rows. There is no third option that is not per-record processing in disguise.
Late, duplicate and out-of-order files are the norm, not the exception. Design the consumer for them.
The small-file problem downstream. Frequent small drops produce metadata pressure and very slow scans in every columnar or table format (Parquet, Iceberg and similar). Compaction (merging small files into big ones) becomes a permanent chore.

Sneakernet (physical disk shipping, such as AWS Snowball) belongs on this list, seriously. Compare bandwidth × time with a truck. Worked example: moving 1 PB (1,000 TB) over a sustained 1 Gb/s link takes 10¹⁵ B × 8 / 10⁹ b/s = 8 × 10⁶ s ≈ 93 days. Past a few hundred terabytes, the truck wins on throughput by about an order of magnitude, while its latency is measured in days. It is the purest example of the latency-versus-throughput trade-off: the two ends are so far apart that they use different vehicles.

The shapes: fan-in, fan-out, broadcast, multicast, anycast

These are not transports. They describe how many senders and receivers there are. Each has one non-obvious consequence.

Fan-out (one sender, many receivers). The failure mode is partial success: three of five subscribers got the record. There is no atomic multi-destination delivery unless you use a transactional outbox (a table written in the same database transaction as the business data, and then published reliably; Transactional outbox) or a shared log that all five read. That is why the shared log is architecturally cleaner: it turns fan-out into a read-side concern, and reads can be retried independently.

Fan-in (many senders, one receiver). The failure mode is ordering. Merging K ordered streams into one totally ordered stream needs either a single serialization point (which caps throughput at one writer) or a comparable clock plus a willingness to wait for the slowest input. That wait is the low watermark ("no record older than time T will still arrive"), and it is the literal, measurable price of order: your end-to-end latency becomes the latency of your laggiest source. Fan-in is also where hot spots and load shedding first appear, because N producers converging on one consumer is the classic overload geometry.

Broadcast / multicast (one send, N receivers, with copying done by the network instead of a broker). A real use is exchange market data, where the broker's fan-out cost is unacceptable at microsecond latencies. The trade is total: IP multicast uses UDP, so there is no reliability, no ordering guarantee across the network, and no persistence. Production systems add gap detection with sequence numbers plus a retransmit-request channel over TCP, which means they built a reliability layer anyway, just one tuned for the 99.99% case. This is not relevant to most data engineering. It is useful to know it exists as the extreme corner of "no durability, minimum latency".

Anycast is not a data-movement pattern at all. It routes to the nearest of N identical endpoints, so it is a placement mechanism for DNS, CDNs and load balancers. It belongs with routing, not transport, and this mechanism does not treat it as a transport.