Summary and key takeaways
The one sentence. A format is a treaty, a protocol is a conversation with amnesia, an integration is a bookmark. The only thing you ever choose is who does the interpreting, and how many times.
The reframe. At a boundary, producer and consumer cannot be updated atomically, and the producer may be gone. Efficiency stops being the deciding vote. What decides is the number of parties who can take part without negotiating. Formats win on installed base, not on quality.
One variable, three families
| Family | Time gap | Back-channel | Contract lives in | Hard problem |
|---|---|---|---|---|
| File formats | Unbounded | None | The bytes, or a registry agreed earlier | Where does the schema live? |
| Access protocols | Short | Full | The conversation | What if the conversation dies? |
| Integration patterns | Indefinite | Yes, with memory needed | Durable external state | What is the position, and is it total and monotonic? |
Key facts, by part
- Files. The schema lives nowhere, in the bytes or in a registry. The parse tax is paid by the reader, forever. Splittability is a framing property: NDJSON gets it for free, CSV with quoted newlines does not, and one gzip stream blocks it. Columnar files need a footer, so they cannot be streamed, and the footer is also a query interface. Zero-copy formats cost total agreement. Table formats are specified metadata protocols. Logs have no ending, so they only append and scan. Six agreement-level hazards (encoding, decimal vs. float, null/empty/absent, timestamps, integer width, collation) make both sides parse correctly and disagree silently.
- Protocols. You cannot know whether a failed request happened. Declarative SQL lets the callee optimize. Dialects come from capability differences. Row cursors are bad for huge results, so large results leave by unload. Statelessness makes REST scale at the price of round trips. Caller-chosen shapes (GraphQL) make cost unpredictable, so defences re-create endpoints. A winning API exports its limits to everybody. Wire compatibility is cheap, semantic compatibility is not. Webhooks force duplicates, so receivers must be idempotent. A protocol must tell callers whether a retry is safe, must publish its limits, and must plan for versioning cost (dual-running × callers).
- Integrations. Position must be durable state. A source is only as resumable as its position is total and monotonic. ELT versus ETL is reprocessability, flipped by economics. Federation depends on pushdown. Mounting leaves maintenance with nobody. Zero-copy sharing is tighter than a copy. Reverse ETL imports analytical delay into operational systems. Paginate by value, not by count. File drop rebuilds protocols from convention: the filename is the protocol, the manifest is the commit, and absence needs a clock to detect.
Three equations and rules of thumb
| Rule | Meaning |
|---|---|
| Cost of change = dual-running time × number of counterparties | Why mature interfaces calcify |
| N × M integrations without a standard, N + M with one | Why drivers, connectors and shared layouts exist |
| Pushdown gain = full transfer ÷ filtered transfer | Why federation succeeds or fails (1,000× in our example) |
Patterns that repeat
- Early binding moves cost off the read path and onto the agreement.
- Costs are conserved: the question is which side pays and whether it can refuse.
- Manifest plus atomic commit keeps returning wherever a set of objects must become visible at once.
- Zero-copy through total agreement appears on files, then on the wire.
- Failures at a boundary are silent: the consumer pays, later.