Chatty requests, needless recompute and cross-region calls
Why do small files, full rebuilds and cross-region calls survive code review?
Chatty small requests
The file count follows
and every factor is set by someone who is optimizing correctly for something else: parallelism, pruning granularity, ingest latency. Worked example: 50 tasks × 24 partitions × 120 commits per hour = 144,000 files per hour. The producer optimizes for write-side concerns, and the consumer pays per request, forever, on every read. This is cost relocated across a party boundary, generated by three independently correct decisions.
It is not self-correcting because the producer's metric improves (low ingest latency) while the cost lands on a different team's bill, under a different meter.
Needless recompute
A full recompute is chosen because it is idempotent by construction: run it again and the answer is right, with no change-detection machinery and no correctness liability. It is buying determinism with money. That is a defensible trade at small scale and ruinous at large scale, and the break-even moves as the table grows while nobody revisits it.
This is a design whose sign flips at a scale boundary that it crosses silently. The organizational mechanism is the important part: a daily full refresh that was correct at 1 GB and absurd at 10 TB never triggers a review, because it never fails.
Cross-region chattiness
A region is a configuration string in your code. Crossing one changes latency by about two orders of magnitude and adds a per-byte charge, yet nothing at the call site makes the boundary visible.
It has the same root cause as SELECT *: an abstraction that deliberately hides topology meets a meter that prices topology.