Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

GRPC, Arrow Flight, the S3 API and the Postgres wire protocol

What do RPC frameworks and de facto protocols buy, and why is protocol compatibility not semantic compatibility?

gRPC and Arrow Flight

These are two different things, and it helps to separate them.

Generic RPC: early binding of the interface

An RPC (remote procedure call) framework such as declares the contract in a definition file. The definition is checked at build time and generates code for both sides. Failures move from run time to compile time. This is the opposite of SQL's placement: early . (What a caller does when a call times out, deadlines and retries, is coordination's question.) The cost: both sides must share the definition, so the interface is a coupled artifact, and every change enters the "no change" situation of Principle A.

It also gives streaming in both directions, which restores flow control: a stream can slow the sender (), while a plain request and response cannot.

Columnar RPC: zero-copy lifted to the wire

If both sides already use the same in-memory layout (), the transport can carry that layout directly. Then the whole serialize-and-parse step disappears from a network path rather than a file path. Arrow Flight does this.

Notice what happened: through total agreement has moved from files to the wire.

It appears exactly here for a reason. It is worth doing only after the bottleneck has moved: when the network is fast and the data is columnar, is the biggest term left. The fixed cost keeps moving upward as the parts beneath get faster, and each time it arrives somewhere new, a mechanism is invented to attack it there.

In one sentence: generic RPC buys compile-time safety at the cost of coupling; columnar RPC buys zero parse cost at the cost of agreeing on memory itself.

The S3-compatible API as a de facto storage protocol

S3 is Amazon's object storage service. The mechanism worth understanding is not the API. It is how an implementation detail of one vendor became a base that everyone reimplements.

The causal chain:

  1. Many tools were written for one interface.
  2. Rewriting the tools is expensive.
  3. So implementing the interface became cheaper than persuading the tools to change.
  4. So competitors implemented it.
  5. So the interface acquired an installed base larger than any implementation of it.

This is Principle B in full form: adoption compounds until the interface outlives and outranks its owner. The followed the same path, and you should expect it wherever a tool ecosystem is large and the interface is narrow enough to copy.

Its limits become industry limits

The second-order effect matters most: the interface's limitations become constraints for the whole industry. Examples:

  • no partial in-place write (you replace the whole object),
  • a per-request price,
  • a minimum per request,
  • metadata behaviour that is only eventually consistent in some implementations,
  • scoped to key prefixes.

These are not laws of storage. They are properties of an API that won. The table-maintenance jobs that data teams run, and half of what a cloud bill charges for, exist because of those choices. It is easy to confuse an economic accident with a physical constraint. It is an accident. It is also, today, effectively permanent, which is a different thing from being necessary.

In one sentence: , and they then look like physics.

The Postgres wire protocol as a compatibility mechanism

This is the same principle, used on purpose in the other direction. A new system adopts an old system's protocol to inherit its client ecosystem: drivers, BI tools, ORMs (libraries that map objects to tables), every dashboard anyone built. It buys instant distribution at zero integration cost to users.

What it cannot buy is meaning. Speaking the wire protocol guarantees that clients can connect and send bytes. It guarantees nothing about the isolation level, type behaviour, transaction semantics or which functions exist. So compatibility is real at the transport level and partial above it. The gap stays invisible until something subtle breaks: a tool expects a certain table, or expects a type to convert in a certain way.

This is a general rule of compatibility layers:

Again: an assumption that cannot be verified, and a who pays.

In one sentence: speaking someone's protocol gets you their clients, not their behaviour.