Per-request pricing: the envelope and the letter

Below what object size do you pay for the request rather than the data?

Cost per useful byte for object storage is

where S is the object size, c_req is the fee per request and c_byte is the cost per byte read or scanned. The request fee dominates when S is below S* = c_req ÷ c_byte.

S* = 4 × 10⁻⁷ ÷ 5 × 10⁻¹² = 8 × 10⁴ bytes, which is close to 10⁵ bytes (about 100 KB).

Below about 100 KB you pay for the envelope (the request) and not for the letter (the data). At 10 KB the request term is 4 × 10⁻⁷ ÷ 10⁴ = 4 × 10⁻¹¹ per byte, against 5 × 10⁻¹² for the data itself: you pay close to ten times what the bytes alone would cost.

Where reading bytes inside a region is free, the byte term goes to zero and the request fee dominates at every size. Then price no longer limits file size from below. The upper limit on file size comes from pruning granularity (large files cannot be skipped in part) and from parallelism (you need enough files to keep many workers busy).

Two floors that agree

There are two separate lower limits on object size:

• the economic floor: the fee must be amortized over enough bytes (about 10⁵ bytes by the formula above, and larger once other costs are added);
• the physical floor: each request has a minimum latency of 20–100 ms, so small requests waste time.

These two floors land within a factor of ten of each other, both in the neighborhood of hundreds of MB (10² MB) as a comfortable target. That is why the target has stayed stable across three generations of storage. The usual explanation, "128 MB because that was the HDFS block size" (HDFS is the Hadoop distributed file system), is historically true and causally dead: the number persists because two independent forces point at the same place.

LIST is a different meter on the same surface

A LIST call is priced per thousand keys. Its cost scales with how many keys (files) a folder holds, not with how many bytes of data they contain.