End-to-end checksums, scrubbing and bit-rot detection

What must a checksum cover to catch real corruption, and how often must you scrub before the last good copy expires?

One word, three failure models

The word "checksum" is used for three different jobs. They answer three different questions against three different adversaries:

The checksum here must survive indifference, not malice. That is why CRC32C (a fast, simple checksum) is adequate here and completely inadequate against an attacker. Different threat model, different tool, same word.

The failure model, corrected

Folklore says "bit rot": charge slowly leaking from the medium. Field evidence says otherwise. Large studies (CERN's 2007 silent-corruption audit; Bairavasundaram et al.'s multi-year study of about 1.5 million drives) found that the corruption you actually meet is dominated by the path, not the medium:

• firmware bugs;
• misdirected writes: the drive wrote your block perfectly, to the wrong address;
• lost writes: acknowledged, never performed;
• torn writes;
• bad cables and host bus adapters (HBAs);
• memory without error correction (non-ECC DRAM);
• ordinary software bugs.

Rates are of the order of 10⁻⁴ to 10⁻² of drives per year showing at least one silent error. And, critically, errors are strongly correlated within one drive and within one manufacturing batch. That is exactly where redundancy plans that assume independence fail (Durability across machines).

This correction drives the whole design. Media decay would be caught by the drive's own error-correcting code (ECC). Misdirected and lost writes are not caught: the block that comes back is perfectly intact. It is just the wrong block, or the correct-looking stale one. No amount of integrity checking inside one device catches that.

The end-to-end argument, plus an anchor

The checksum must be computed by the party that produced the data and verified by the party that consumes it. And, the part that most designs get wrong, it must cover the block's identity, not only its contents. A checksum stored inside the block validates a misdirected write perfectly, because the block and its checksum travelled together and agree with each other.

Three real designs:

• ZFS stores each block's checksum in the parent block pointer, not in the block itself. A misdirected write cannot bring a matching checksum with it. The tree validates itself from the root.
• InnoDB and Postgres embed the page number in the page and checksum it. That detects misdirection within the file.
• T10 DIF/PI (a standard for data-integrity fields) attaches 8 bytes to each sector: a guard (checksum), a reference tag holding the expected block address (LBA), and an application tag. It is verified at every hop, from the host adapter to the media.

The general rule:

Six forms of the same rule: ZFS parent-pointer checksums, page number inside a page, T10 reference tags, Merkle trees (hash trees where each parent hashes its children), content-addressed stores (data named by its own hash), and table-format manifests (the data file is valid, but only the manifest says it belongs to the table). A seventh appears in distributed systems: fencing tokens, because a leader cannot prove from inside itself that it is still the leader.

Scrubbing, derived from a timing inequality

Latent corruption is found only by reading. Nobody reads cold data. So corruption collects exactly in the data least likely to be read, which is also the data whose redundant copies are aging in parallel, under the same conditions. Hence scrubbing: proactively reading all data and checking its checksums on a schedule.

The design constraint is an inequality that almost nobody writes down:

scrub_period + detection_to_alert_lag  <  retention_period of the last good copy

Cost scales with stored bytes, permanently:

• 100 TB on a 30-day cycle: 100 × 10¹² B ÷ (30 × 86,400 s) ≈ 40 MB/s of background reads. Practically free.
• 10 PB on the same cycle: ≈ 4 GB/s. Real capacity.

The usual answer is to lengthen the period, which lengthens detection lag, which tightens the inequality. There is no escape, only a choice of where to pay: scrubbing is a permanent tax proportional to the stored bytes.

The boundary of what detection buys

A checksum recovers nothing. It turns silent loss into loud loss. That is a real improvement: silent corruption spreads into backups, replicas and derived tables, while loud corruption stops at the reader. But the repair needs a second copy.

This is why ZFS, Ceph, HDFS and Colossus all pair checksums with automatic repair from a replica instead of offering detection alone.

Failure modes of the verifier

• A mismatch cannot tell you, from the inside, whether the data is corrupt, the checksum is corrupt, or the checksum code has a bug. So a system that panics on a mismatch can be taken down by a bug in its own integrity code.
• When two layers both checksum and they disagree, you get the "which layer is lying?" problem. It has the same shape as the unretryable fsync failure from Group commit and fsync: the stack gives you no way to find which witness is at fault.

How the mechanisms of this article sit on three trade-offs