Why small files appear
What makes a table accumulate small files, and what does each mitigation cost?
How many files does a table get per hour? Roughly:
This is a product, not a function of data volume.The small-file problem is caused by partition count and writer count. Volume is nearly irrelevant. That is why it hits hardest on exactly the tables that people think are too small to worry about.
Three mitigations, and each moves the cost
This is the first place where the principle "costs are relocated, never removed" shows up as a choice:
| Mitigation | What you pay |
|---|---|
| Pre-shuffle by partition key so one writer owns each partition | A shuffle (redistributing data between workers over the network) to avoid a rewrite later |
| Lengthen the commit interval | Freshness: new data appears later in queries |
| Auto-compact (a background job merges files) | Background compute, later, with interest |