Skip to content
The Data EngineeringThe Data EngineeringBeta
About
Main content

Vacuum: reachable is not in use

Why can vacuum compute reachability but only guess liveness, and what does the retention window really bound?

Vacuum: reachable is not the same as in use

Vacuum deletes data files that the table's metadata no longer references. Whether a file is safe to delete hides a difficult problem.

  • The metadata says exactly which files each references. If a file is in no retained snapshot, it is unreachable.
  • Liveness is not computable. A reader that opened a snapshot twenty minutes ago holds file names in its own process memory. The table cannot see that. You cannot list your readers, so you cannot know that a file is safe to delete.

There are two exits:

  1. Register readers. Every reader tells the table "I am reading snapshot 42." That needs coordination, which is exactly what object-store-native designs were built to avoid.
  2. Assume a maximum reader lifetime T and delete any file that has been unreferenced for longer than T.

Everyone takes the second exit.

That makes the The default values of 5-7 days are conservative guesses at "the longest plausible reader." If a query runs longer than the window, it fails in the middle of its scan with a file-not-found error.

A read-only report, killed by cleanupwindow 1 h · report 3 h

This is epoch-based reclamation (a technique from in-memory data structures in which memory is freed only after every thread has moved past an "epoch" or time period) with the epoch replaced by a wall-clock guess. It gives the cleanest answer here to two standing questions: the system assumes that no reader is older than T; it cannot verify this; and the long-running reader pays.

The slowest reader

It has four independent sightings: PostgreSQL pinned by a long transaction (MVCC); log retention pinned by a lagging consumer or an abandoned replication slot (truncation and retention); stream-processing watermarks pinned by one straggling source ; and distributed garbage collection. A is the same bound with the reader's lifetime guessed instead of tracked.