File drop, backfill, and what integration forbids

How does a file drop commit without any channel, and why do backfill and steady state need different interfaces?

File drop and partner exchange

This pattern is easy to dismiss, but it still carries a large share of data flow in finance, healthcare, telecom and government. Examples: SFTP (secure file transfer), batch delivery, manifest-and-checksum protocols. It is mechanically different, because there is no back-channel, no catalog and no shared runtime: just files appearing in an agreed place on an agreed schedule.

So everything that protocols provide must be rebuilt from convention:

That last row is the only failure in this whole mechanism that needs a clock to detect. Every other failure produces some signal (an error, a wrong value). A missing file produces nothing.

# Producer side: a safe way to publish a batch
cp part-*.csv.gz /drop/.tmp/                     # write to a staging folder first
sha256sum part-*.csv.gz > MANIFEST.sha256        # integrity list
echo "files=12 rows=4812007" >> MANIFEST.sha256  # control totals
mv /drop/.tmp/part-*.csv.gz /drop/incoming/      # rename is atomic per file (same filesystem)
mv MANIFEST.sha256 /drop/incoming/               # manifest LAST = the commit
# Consumer: ignore any batch whose MANIFEST is missing; verify checksums and totals.

You have seen this shape before: manifest plus atomic commit is the table format, the write-then-publish discipline used for safe data publishing, and the manifest indirection in storage engines. The same solution appears because the same constraint appears: a set of objects cannot describe itself, and visibility must be separated from writing. Here it appears in the most primitive setting there is, which is a good argument that it is fundamental, not fashionable.

In one sentence: with no channel at all, the filename is the protocol and the manifest is the commit.

Backfill versus steady state: two interface regimes

Steady-state synchronization (small, frequent increments) and the initial load or backfill (all of the history, once) have different rate profiles, different failure tolerance and usually different mechanisms. Each of these is its own named problem inside a pipeline. What matters at the interface level is that one interface often cannot serve both. An API sized for incremental polling is the wrong instrument for extracting five years of history, and the provider frequently has no bulk path at all.

What integration forbids

• You cannot have a resumable integration without a durable, totally ordered position.
• You cannot paginate correctly against changing data by counting.
• You cannot push down what the far side cannot compute.
• You cannot remove a copy without inheriting the original's availability and layout.
• You cannot make a set of files complete without a separate act that says so.
• You cannot change a schema across an organizational boundary on your own schedule.

The escapes and their prices