Object storage and the lakehouse
Why is the lakehouse a forced construction?
Block, file and object storage
These are not three products. They are three points on one axis: how much the interface lets you address and change.
| Kind | What the interface gives you |
|---|---|
| Block storage | Addressable by offset in fixed-size blocks, changeable in place. No namespace and no metadata. |
| File storage | Adds a hierarchical namespace and byte-range changes under POSIX semantics. The namespace, not the data path, is what makes distributed file systems hard: directory operations, rename atomicity, locking, metadata servers. |
| Object storage | A flat namespace (prefixes are a display convention, not a structure), immutable whole-object writes, an HTTP interface, and effectively unlimited scale. |
Why object storage won analytics
The derivation is a causal chain, not a feature list: it gave up exactly the operations that need coordination.
- Immutability removes write-write conflicts.
- A flat namespace removes directory locking and rename atomicity.
- No byte-range changes removes partial-write atomicity.
What remains is a system that scales horizontally because there is nothing to coordinate, achieves extreme durability through erasure coding (whose degraded read waits for the slowest of k fragment requests, a tail-latency cost disguised as a durability choice), and costs roughly a quarter of block storage per byte.
The consequence then lands back on this mechanism with full force:
Immutable, plus 50 ms latency, implies large units. Large units make copy-on-write catastrophic at
U/d≈ 10⁵-10⁶. That forces delete vectors. Immutability forces manifest indirection. Indirection forces compaction.
The whole lakehouse architecture can be derived from two properties of an HTTP API. That is the strongest claim in this article, and it holds without qualification.
Two honest corrections
- "Object storage is cheap" is true per byte stored and false per operation (request pricing). At about $0.0004 per thousand GETs, a billion small reads cost 10⁹ ÷ 1,000 × $0.0004 = $400 in requests alone, which can exceed the storage bill. Your row-group size is partly a billing decision. The fixed cost in the batch-size rule is, literally, denominated in dollars here, not only in milliseconds. This is the first time here that the constraint is economic and not physical.
- The "separation of storage and compute" advantage is partly an artifact of how cloud vendors set prices, not a property of the technology. This is an economic contingency. It could change.
Parcelo's version. This explains her bill. Her first prototype (Data at rest and the one ratio) kept 4 KiB files: 400 GB ÷ 4 KiB ≈ 100 million objects. Each full scan then costs 10⁸ ÷ 1,000 × $0.0004 = $40 in GET fees, per scan. With 128 MB files (3,125 objects) the same scan costs about 3,125 ÷ 1,000 × $0.0004 ≈ $0.001. The prices in this example are illustrations, but the ratio (about 32,000×) is the point.
A cache in front of the store
A small fast tier in front of object storage captures much of the benefit when access is skewed. How much it captures, why its misses dominate the mean, and why one full scan can evict its working set are caching's subject (with scan resistance).
The LSM tree and the lakehouse: one forced construction
Assemble the ingredients, and it falls out with no design freedom:
- a medium with a high fixed cost per write and no in-place update,
- an in-memory buffer,
- immutability.
Then:
- The memtable (the in-memory buffer) is write coalescing, the same batching as group commit.
- The flush size is the natural batch of the batch-size rule.
- The SSTable (the immutable sorted file) is the immutable unit. It carries its own layout and clustering.
- Compaction is the garbage collector of the immutable-substrate rule, with the policy arithmetic from Mutation strategies.
- The read path is merge-on-read.
Every component is a mechanism you have already derived on its own, and their composition was forced. That is why the LSM tree keeps being reinvented. It is what you get, not what you choose.
The lakehouse is the identical structure, one level up. Object storage shares NAND flash's defining properties, so it grows NAND's organs: data files under manifests, under a manifest list, under a metadata pointer.
Two things about this structure that marketing does not say:
1. The atomicity does not live in the object store. There is no atomic multi-object operation in S3. So the whole transactional story reduces to a single compare-and-swap (CAS: change a value only if it still equals what you read) on one pointer. That CAS lives in a catalog: DynamoDB, a metastore or a relational database. "Transactions on S3" is a phrase that moves a coordination point and does not remove one. The catalog is the real single point of coordination, with the availability and contention properties of whatever it actually is. It is the same reduction as in copy-on-write, with the same trick at a different scale.
2. Planning has a latency floor. Metadata file → manifest list → manifests → data files is three or four sequential round trips before the first data byte. At about 50 ms each, that is a floor of about 4 × 50 ms = 200 ms on every query, whatever its size. So these systems are excellent for scans and unusable for point lookups. This is the batch-size rule at the metadata level: the fixed cost is amortized only by large results, so the architecture has a minimum viable query size, just as it has a minimum viable file size. (This is one more reason the tracking page, in the story, stays on a key-value store.)