Scenarios

If the file has quoted newlines, or is one gzip stream, a single worker reads it: 12,800 MB at about 100 MB/s is 128 s of pure parsing, against about 1.3 s with 100 blocks of 128 MB in the ideal case.

You cannot easily make the carrier change. Land the file, convert it once to a splittable columnar format (Parquet), and let every later reader pay the cheaper parse cost.

Encoding, decimal rules, null rules and time zones; and add a check against control totals, because type inference can break at row 4,000,000.

One batch conversion step and storage for two copies, in exchange for parsing once for all readers. Ask the carrier for NDJSON or block-compressed files, but treat it as a bonus.

A binary row format, with the schema in a registry and a small schema ID in each message. A 5-byte ID instead of a 2,000-byte schema carried in each message means 50 GB instead of 20 TB per ten billion messages, 400 times less.

Identify fields by number (as Protobuf does), so renames are free, and ban reusing numbers.

The registry becomes a critical dependency: replicate it, cache schemas in clients and back it up, because losing it makes every message undecodable. With 40 teams, changes cannot be made at once, so agree a compatibility rule and a sunset period (cost = time both versions run × teams).

Files archived long-term should carry their own schema, since they may outlive the registry.

The client times out when the reply is slow or lost and retries, but the first request had actually succeeded. The client cannot know whether the remote action happened, so retries create duplicates.

Send an idempotency key with every charge; the server stores the key with the result and returns the stored result on a repeat. Retry by error class: retry 429 and 503 with backoff, never retry 400 or 404, and handle partial success in a batch per item.

The server must store keys and decide how long to keep them, and the key store must be durable and written in the same transaction as the charge.

A daily reconciliation of counts and sums, a check that does not depend on any error being raised.

At 200,000 rows per second, 100,000,000 ÷ 200,000 = 500 s, and a slow client makes it worse while the server holds the session and its memory.

Unload: the engine writes Parquet files to object storage and gives the client a list of files, so the cost no longer depends on the client's pace.

Columnar RPC (Arrow Flight), if both sides support it, which removes converting columns to rows but keeps the live connection.

Unload adds latency and a third storage location that needs cleanup and access control. The cursor stays fine for small results.

A full push of 2,000,000 customers takes 200,000 s, about 2.3 days. Send only changes: if 20,000 customers changed, that is 2,000 s, about 33 minutes.

Sending only changes needs reliable change detection: change data capture, or a position that is total and only increases.

Writes to the CRM are not atomic, so make each record's update idempotent ("set", not "add"), store the position durably, and advance it only after the CRM confirms. Plan for partial syncs (a state that is neither old nor new), backoff against the rate limit, and a dead-letter list.

The warehouse is late and best-effort, so putting it in the serving path imports that delay. The CRM's rate limit, not you, sets the schedule.

The backfill alone takes 400,000,000 ÷ 500 = 800,000 s, about 9.3 days, with no failures and no limits hit.

1. Ask for a bulk export first: one set of files with a manifest and control totals.
2. If there is none, run the backfill as its own regime, with checkpointed keyset pagination, never OFFSET, using a unique tiebreaker such as (ts, id).
3. Then switch to incremental sync with a token or a high-watermark.

Losing an opaque sync token means a full resync; out-of-order writes break timestamp watermarks; offset pages silently skip or repeat rows under writes.

Loading raw first (ELT) means a transformation bug can be fixed without calling the API again.

There is no channel back to the partner, so:

• the filename carries the date and a sequence number;
• the partner writes under a temporary name and renames atomically;
• a checksum file gives integrity;
• a manifest, written last, is the commit, and lists the expected file count, row count and sum of amounts.

Ignore any batch without a manifest, and verify checksums and totals.

You need a clock: a schedule that raises an alert if the batch is not complete by an agreed time.

Encoding, decimal not float, null rules, time zones.

Partial uploads, flipped bytes, skipped files, and the silent missing file.

One truth and one freshness, but a tighter relationship: the partner reads your live layout, so you cannot restructure without breaking people you may not know; their query load lands on your storage bill; and it only works within one catalog and storage domain, so across clouds or vendors you copy again.

Independence and a clear contract (a manifest and a documented schema), but a second dataset with its own freshness and drift.

If the partner is inside the same trust and billing domain and the layout is stable, share by reference, with published limits and a versioning policy. Across organizations, publish versioned exports with a sunset policy. Either way, name who owns table maintenance.