Fsync, flush barriers and O_DSYNC

Why must a failed fsync never be retried?

This is where the ladder's honesty problem lives, and where most real durability failures actually happen.

When write() returns, the data is in the page cache. That is rung 2. It survives kill -9 (killing the process) and nothing else. fsync() is supposed to take you to rung 4, by writing back the dirty pages and sending a cache-flush command to the device. Between the promise and the reality there are two structural lies.

Lie one: O_DIRECT is not durability. The O_DIRECT flag makes a write bypass the OS page cache. It does not flush the device's volatile write cache. O_DIRECT without O_DSYNC (a flag that makes each write wait until it is durable) or an explicit flush is rung 3. This is common folklore, and it is wrong.

Lie two: a failed fsync may report success when you retry. On Linux, when background writeback fails, the error is reported to whichever file descriptor asks first, and the dirty pages may be marked clean anyway. So a second fsync() returns 0 while the data is simply gone. Postgres found this out in 2018. The correct response, which Postgres now follows, is to PANIC (stop the server) on any fsync failure and recover from the log, because there is no sound way to retry. If your code treats fsync failure as retryable, it has a data-loss bug. This is the sharpest answer to "what does this system assume that it cannot verify?": the whole stack assumes that a failed write is still present so it can be rewritten, and it is not.

# WRONG: treating fsync failure as retryable
while True:
    try:
        os.fsync(fd)
        break
    except OSError:
        time.sleep(1)       # the dirty pages may already be marked clean: data lost

# RIGHT: the only safe response is to stop and recover from a durable log
try:
    os.fsync(fd)
except OSError:
    panic_and_recover_from_log()

Two precision points

"Barrier" is the wrong mental picture. Linux removed barrier semantics in version 2.6.37 and replaced them with explicit flags, REQ_PREFLUSH (flush the device cache first) and FUA (Force Unit Access: this one write must reach the media before it is acknowledged). Barriers ordered the whole queue and caused too much serialization. Flush-and-FUA lets the device keep reordering everything except the one thing you named. The general lesson: state the constraint you actually need, not a global ordering that implies it. This lesson recurs in memory models and stream processing.

fdatasync is cheap only on a file that is not growing. It skips metadata that is not needed to read the data back. But the file's size is needed, so an appending write still drags a metadata update into the flush. This is exactly why Postgres pre-allocates 16 MB WAL segments and InnoDB uses fixed-size redo files: the log becomes a sequence of overwrites into already-allocated space, not appends, so every flush is pure data. This one design choice removes one metadata write and one journal transaction from every commit. A small implementation decision changes the shape of the cost, not just its size.

The device may lie as well

Consumer drives may acknowledge a FLUSH command without honouring it. Virtualized block layers may add their own caches. This is the honest form of the central problem:

This is the durability version of the end-to-end argument (only the two ends of a path can check that the whole path worked). It is the reason end-to-end checksums (When the storage lies) exist at all: they are the only mechanism here that checks instead of assumes.

The settings that pick the rung

Every engine lets you choose the rung, and the API does not tell you which one you got. A commit that returns success has said "yes" at whatever rung one line of configuration chose:

A relaxed setting is sometimes the right trade, for data you can rebuild. The failure is not choosing it; it is choosing it without knowing, through a default, a wrapper library or a benchmark tune that was never reverted. Replicas add rung 5 with their own settings (Durability across machines), and a message broker has its own as well (Durability across a pipeline).

How logs, undo, group commit and fsync sit on three trade-offs

The consequence for the rest of this mechanism

So far the engine has all the mechanisms that write durability information, and none that read it back. The reading side is recovery, which appears in Bounding the log right after checkpointing, because a checkpoint has no meaning except as a control on recovery. And two more things are needed: backups (a time-separated copy, Surviving your own mistakes) and independent failure domains (Durability across machines).