Optimal file size: a multiple of F × b

What file size minimizes scan time, and what stops files from growing larger?

Let us derive the best file size from first principles.

A scan reads D bytes of data, split across N files. Each file costs a fixed per-object cost F: the request round trip, reading the file footer (the metadata at the end of a Parquet file that describes its layout), and a metadata entry. On an object store F is about 20-50 ms and it does not shrink when the file is smaller. Then the data itself streams at bandwidth B. So the total cost is roughly:

where P is the parallelism (how many files are fetched at once). The first term is overhead. For one file of size s streamed at rate b per connection, the share of time lost to overhead is:

With F = 50 ms and b = 100 MB/s, a 5 MB file loses half its time to overhead, a 50 MB file about 9%, a 500 MB file about 1%. This is the natural transfer size of the medium, the fixed cost divided out.

Returns diminish quickly. After about 100 MB there is little more to gain. This is the whole reason the industry's target file sizes cluster in the 128 MB to 1 GB band. Nobody chose that number. It is a multiple of F × b (5 MB here), set by how much overhead you accept.

Why not make files even larger?

Because skipping, parallelism and copy-on-write all work per file, a file can also be too large: a file that matches every predicate is always read, too few files leave workers idle, and a one-row update rewrites the whole file. Where those bounds sit is a layout decision; maintenance restores files toward the size it chose.

A correction about small files and money

A common claim says small files are expensive "because of per-GET charges." In dollar terms this is mostly wrong. Worked example: a 1 TB scan with 10 MB files is 1,000,000 MB / 10 MB = 100,000 GET requests. At a typical price of about $0.0004 per 1,000 GETs, that is about $0.04: a few cents. The real damage is latency and, more importantly, planning cost, which we meet in Reclamation under manifest rewriting. Which frame applies depends on what you count: per query the request fees are a few cents, but summed over a year of a fleet's queries they can outgrow the storage bill (see Target file size).