Request/response and RPC

Where does a synchronous call belong in a data platform?

RPC (remote procedure call) is calling a function on another machine and waiting for the answer. HTTP calls to a REST service and gRPC calls are RPC.

RPC has zero decoupling on all three dimensions. The caller blocks (or holds a continuation, meaning code that resumes later), requires the callee to be up right now, and must know the callee's address.

In exchange it gives you the one thing no asynchronous transport can: the result, and an error attributable to this specific call.

What synchronous calls buy, honestly

• Backpressure for free. The caller is blocked, so it cannot run faster than the callee. This is not a mechanism you build; it is a consequence of the shape. It is truly under-appreciated: the most reliable flow control is a blocked thread.
• Errors are immediate and attributable. No DLQ, no correlation ID, no "which of the 40 records in this batch failed?"
• No extra infrastructure. No broker to run, size, upgrade or lose.
• Read-your-writes. After the call the caller knows the new state.

What it costs

• Availability multiplies. A chain of five services that are each 99.9% available is 0.999⁵ ≈ 0.995, which is 99.5%: about 3.6 hours of downtime per month instead of 43 minutes. Every hop is a mandatory dependency.
• Latency adds up, and tail latency compounds. If each hop's p99 is independent, a five-hop chain's p99 is much worse than five times the median. Fan-out makes it worse: waiting for 10 parallel calls means waiting for the slowest, so your p50 becomes their p90.
• Retry storms and metastable failure. A slow dependency makes callers time out and retry, which adds load, which causes more timeouts. The system stays broken even after the original trigger is gone, because it has found a stable bad equilibrium (this is called metastable failure). That is why retry budgets, jitter (random delay) and circuit breakers exist, and why "just add retries" makes outages worse.
• The lost-ack ambiguity. The request succeeded, but the response was lost. The caller cannot tell "not done" from "done, but the ack was lost". There is no protocol fix for this. The only resolution is an idempotent receiver keyed on a caller-supplied idempotency key. This is the same problem that at-least-once queues have. Synchronous calls do not escape it; they only hide it better. (This is Ravi's double charge.)

The RPC variants

Synchronous unary (one request, one reply): everything above.

Async / callback / future RPC decouples the caller's thread from the call, not the caller from the callee. The availability coupling is unchanged. It is a concurrency optimization, not a change of transport. Do not let the word "async" make you think you bought decoupling.

Streaming RPC (gRPC server-streaming, client-streaming, bidirectional) is the interesting one. HTTP/2 (the protocol under gRPC) has flow control through connection and stream windows: the receiver announces how much data it can accept, and the sender cannot exceed it. This is credit-based backpressure built into the transport, so rate is genuinely decoupled, up to the window size.

But availability and lifetime are not. The stream lives and dies with the connection. There is no store. If the receiver dies in the middle, everything in flight is gone, and there is no position to resume from unless the application built its own checkpointing on top.

The clean way to remember it: streaming RPC is a queue with zero durability and a lifetime tied to one connection. It is the right tool for continuous transfer between two live processes that can restart from a known point (replication, log tailing, model-inference streams). It is the wrong tool whenever "the consumer was down for 20 minutes" must be survivable.

Where synchronous calls belong in a data platform

In the control plane, not the data plane. Trigger a job, check a status, fetch a schema from a schema registry, resolve a lookup. The moment you move records at volume, a synchronous call per record gives you the worst position on latency-versus-throughput (a network round trip per record) and the worst availability coupling. If you must call a service once per record, batch the calls, so that you trade latency for throughput deliberately and not by accident.