NVMe, persistent memory, tape and object storage

What do NVMe, persistent memory, tape and object storage each change in that answer?

NVMe

NVMe is the modern protocol for talking to SSDs over the PCIe bus. It was designed for flash, not inherited from spinning disks.

The interface change (many deep queues, one per CPU core) is the small part. The big point: one NAND chip is slow. The drive is fast only when many chips work at once, across 8–16 channels with several dies each.
So Little's Law governs everything: Concurrency = Throughput × Latency.

Worked example (Meera's 12,500 IOPS)

One NVMe read takes ~80 µs. If you send one request at a time (queue depth QD = 1), you get 1 / 80 µs = 12,500 IOPS. No hardware can change this. To reach 1,000,000 IOPS you need 1,000,000 × 80 µs = 80 requests in flight at the same time. Most "my NVMe is slow" reports are QD = 1 benchmarks.

IOPS and bandwidth are two separate limits. Small requests hit the IOPS limit; large requests hit the bandwidth limit. They cross at about 16–64 KB. Vendors quote both; you never reach both together.
Past saturation, extra concurrency only adds tail latency (the queue formula R = S/(1 − ρ), Performance laws).
Software becomes the fixed cost. At 1,000,000 IOPS per core you have ~1 µs per I/O, but the operating-system I/O path costs 3–10 µs. Cause 3 moved up one level: now the kernel is the bottleneck. This is why io_uring (a Linux interface that submits many I/O requests with few kernel calls) and SPDK (user-space NVMe drivers that skip the kernel) exist.

Persistent memory

Persistent memory is byte-addressable like RAM but keeps its contents without power. Intel Optane was the main product.

The interesting point was never speed. Persistence and cache coherence are different boundaries. When your program stores a value, it first lands in the CPU's volatile caches. It is not durable until it reaches the persistent medium. So you need an explicit flush + fence (instructions that push data out of the cache and wait until done), or a platform feature (eADR) that flushes caches when power fails.
Only 8-byte stores are atomic (happen fully or not at all). Anything larger needs logging or shadow copies to survive a crash in the middle. Byte-addressable durability does not give you crash-safe structures for free. It gives you the building block and the duty.
Intel stopped Optane in 2022. That is a Cause 6 death, not Cause 0: good physics, bad economics. The idea continues in CXL (Compute Express Link, a fast interconnect for attaching memory). With +170–250 ns latency it is a new tier between RAM and storage, not a replacement for either.

Tape and cold tiers

Tape is the bottom of the ladder: one head serves kilometers of ribbon. Lowest cost, coarsest granularity (Cause 2). Everything about tape follows from that ratio.
A common myth: real online-tape retrieval takes minutes (mount the cartridge, load, wind to position). The 12–48 hours advertised for cloud archive tiers (such as AWS Glacier Deep Archive) is a batch-scheduling policy: the provider groups requests to use drives efficiently. It is not tape physics.
Shoe-shining is Cause 3 in its purest form. If you cannot supply data at the drive's native speed, the drive must stop, reverse and reposition, again and again. Throughput collapses and the tape wears. So data is always staged on disk first, then streamed to tape at full speed.
Tape's real value is not price per TB. It is offline by default (a real air gap: ransomware cannot reach a cartridge on a shelf) and it fails gracefully. The hidden cost is generation migration: every few years the drives change and the data must be copied to new media.

Object storage

Object storage means cloud services such as Amazon S3, Google Cloud Storage and Azure Blob. You store "objects" (files with a name, called a key) through HTTP calls. Most modern data lakes sit on it.

The medium is a network service, so the fixed cost is a round trip, not a machine. Time to first byte (TTFB) is 20–100 ms and cannot be reduced: DNS lookup, TLS handshake, authentication, metadata lookup. Cause 1 (distance and service depth) dominates.
The natural transfer size is roughly 8–100 MB. Worked example (Meera's nightly job): reading a Parquet file as 1,000 small requests costs 1,000 × 50 ms ≈ 50 seconds of pure waiting. The fix is Little's Law: combine small reads into a few large range GETs (HTTP requests for a range of bytes), and run 20–100 of them in parallel.
No partial in-place write. You cannot edit byte 500 of an object; you replace the whole object. So, once more: indirection layer + GC. Iceberg is an FTL for object storage. Its manifests are the mapping table (which files form the table now). Compaction and snapshot expiry are the garbage collector. This is the same structure as flash, with latency about three orders of magnitude higher. It is probably the most useful analogy in this whole field.
Strong consistency arrived in December 2020 (for S3), but only per object. After a successful write, reading that object returns the new data. But there is no atomic change across objects and no atomic rename. Renaming a "folder" copies every object under that prefix one by one, and it can stop halfway. So every commit protocol on object storage reduces to: get one atomic pointer swap, and make everything else safe to repeat (idempotent).
Per-prefix rate limits exist because the metadata layer is itself a partitioned distributed system. When traffic to one key prefix grows, the service must split that partition. This takes ~30–60 minutes, so a sudden burst receives 503 Slow Down errors while it adjusts. The key space is split in alphabetical order, so request capacity follows key distribution, not bucket size: a date-leading key (2026-10-05/shop-1043/event-8841.parquet) puts every reader and writer of the current day into one contiguous range. The fix is entropy near the front of the key (a3f9/2026-10-05/…, with a3f9 a short hash of the shop id), which fights human readability and prefix-based pruning, another interior optimum between two pressures on one granularity. And prefix hotspotting was not "fixed": it was made adaptive, with a time constant of minutes to hours, so the failure moved out of steady state into transients (new partitions, backfills, cutovers), the moments you are least able to slow down.
Cost is a physical law here. Per-request pricing punishes small files three times: request fees, the latency floor, and metadata overhead. The "small-files problem" was never only a Hadoop issue.

Same four questions, five answers

	Read unit	Write unit	Reclaim unit	Fixed cost	Concurrency needed	In place? → indirection
HDD	4 KB sector	4 KB sector	—	~13 ms mechanical	1–4	Yes (SMR: no → zone map)
NAND	page 4–16 KB	page, in order inside block	block 4–24 MB	~500 µs program / ~5 ms erase	~64–128	No → FTL + GC
PMEM	64 B cache line	64 B (8 B atomic)	—	~300 ns	low; worse above ~4 threads	Yes, but flush needed
Tape	~MB	sequential only	whole tape	30 s – 4 min	1	No → rewrite
Object	byte range	whole object	object	20–100 ms	20–100	No → manifests + GC

Read the last column from top to bottom: every medium that cannot update in place grew the same architecture. When you meet a new storage medium, ask question 4 first, then look for its mapping table and its garbage collector. They exist.