Interfaces & Formats
What can two parties agree on when they cannot talk to each other?
Who this mechanism is for: a developer who writes code, reads and writes files, calls APIs and knows basic SQL, but has never studied why file formats, wire protocols and data-integration tools look the way they do. You do not need any database or systems background. Every technical term is explained the first time it appears, and again in the glossary at the end. Where an idea from storage, queues or networks is needed, it is explained again in a few sentences on the spot.
The story we will follow
Priya is a data engineer at Harborline, a freight-forwarding company. Harborline books cargo on trucks and ships that belong to other companies (the carriers). Carriers send Harborline shipment files every night. Harborline loads them into its own analytics tables, offers a partner API, and pushes customer data into a sales tool. Priya's colleague Tomas runs the integrations with partners. Harborline controls its own code, but it does not control the carriers, the sales tool, or the customers' spreadsheets.
During one busy quarter, Priya will meet five problems, one at the start of each article below. Each problem looks like a bug in Harborline's code. Each one is actually a consequence of one fact: the two sides of an interface cannot be changed at the same moment, and they cannot always ask each other questions. At the end of each article we go back and solve the problem.
This is the main idea of this whole mechanism:
How this mechanism is organized
- 1
The boundary
Why does the deciding rule change from "fastest" to "what can both sides agree on"?
CaseA rename that broke a carrier for three weeks
- 2
File formats
When the writer may be gone and nobody can ask a question, where does the schema live, and what does each format cost the reader?
Case100 workers, and one does all the work
- 3
Access protocols
When a live conversation is possible, what moves into the conversation, and what happens when the conversation breaks halfway?
Case100 million rows through a driver, and a double charge
- 4
Integration patterns
When a relationship lasts forever, why must the "position" become stored state, and who pays for each design?
Case214 customers missing, and no error logged
- 5
What interfaces forbid
What are the three questions behind every interface, and what can no interface give you?
CaseWhy did the customers find them first?
After that: a summary, hands-on labs, interview questions and a glossary.
One request: do not skip the small examples with bytes, rows and dates. Most interface defects are small things (an empty string, a time zone, a page boundary). Seeing them once makes them easy to recognize later.
What do I do, and what does each option cost?
Can I see it myself?
Splittability: CSV with quoted newlines versus NDJSON
Split a CSV with quoted newlines and an NDJSON file at random byte offsets and count the records each split breaks.
Break itPythonThe footer pays for column skipping
A 200-column Parquet file: read 3 columns against all 200, read the footer statistics, then append bytes after the footer.
MeasurePython with PyArrowThe six interop hazards
Write values one way and read them another: encoding, decimals as floats, null/empty/missing, zoneless times, int32 overflow, collation.
Break itPython with NumPyIdempotency: a webhook that retries
A charging server that is sometimes slow and a client that retries on timeout: charges with and without an Idempotency-Key.
Real systemPythonOffset versus keyset pagination, under writes
Page through a million SQLite rows with OFFSET and with a keyset predicate: time at page 10,000, and duplicates after one insert.
MeasurePythonFile drop with manifest as commit
Publish file batches with stage, checksum, atomic rename and manifest last; break them three ways; then delete a batch with no schedule.
Break itPython
Can I reason about it under questioning?
Interview questions
Short questions on interfaces and formats, such as file formats and splittability, schemas and registries, JDBC, REST and GraphQL, webhooks and retries, connectors, federation and pagination, each with an answer you could give aloud and a link to the full explanation.
Scenarios
Longer problems on interfaces and formats, of the kind asked in senior interviews: each one estimated with numbers, diagnosed, and answered with its trade-offs.
The tensions this mechanism takes a position on. Each recurs across mechanisms.
Early binding vs. late binding
When structure is committed; determines where the failure surfaces, at write time or read time.
Read-optimized vs. write-optimized
For example: B-tree vs. LSM, merge-on-read vs. copy-on-write.
Coupling vs. coordination cost
For example: 2PC vs. sagas, schema contracts vs. schema-on-read.
Push vs. pull
Who initiates the work.
Ideas this mechanism meets that keep coming back, mechanism after mechanism.
Costs are conserved
Costs are conserved, never removed.
Absence is not evidence
Absence of an event is not absence of a failure.
Designate one write
Atomicity over n items is atomicity over one designated item.
Undeclared dependencies
An optimization that succeeds becomes an undeclared dependency, with leverage 1/(1 − h).
Coordination cannot be made cheaper
Coordination is the only thing you cannot make cheaper.