Retries, the unknown state and fencing

Why can no orchestrator tell a dead task from a slow one, and what does fencing need from the sink?

Retries, the task state machine and fencing

Retry policy

Retry policy itself is familiar material: backoff with jitter, a cap on attempts, and a retry budget so a systemic failure does not become a retry storm.

The orchestration-specific part is what the retry is reacting to.

The state machine has an unknown state

A task passes through queued → running → succeeded or failed. But there is a fourth position: running according to the database, but unreachable in fact. Heartbeats (small "I am alive" signals sent every few seconds) have stopped. The orchestrator cannot tell whether "the process died" or "the process is fine, but the network or the heartbeat writer is not."

This is not a gap in today's tools. In an asynchronous system you cannot tell a crashed node from a slow one (failure detection), and a task that is neither clearly up nor clearly down is gray failure arriving in the control plane.

And the orchestrator must act anyway. Both choices are wrong in some case:

Assume dead, relaunch. If the original is alive, two processes now write the same output. Even with an idempotent overwrite, their writes may interleave and produce a torn partition: half from each copy, both "correct", the combination neither.
Assume alive, wait. If the original is dead, the pipeline stalls forever and the SLA is missed with no failure signal. This is worse than a failure, because failures raise alerts.

Two correct attempts, one torn partitionattempt 7 paused · attempt 8 relaunched

overwrite in place

visible: half a8, half a7 — torn

write-audit-publish

pointer swap to a8 wins; a7's swap fails: visible = all a8

Fencing

The resolution is not better detection, because detection is impossible. It is fencing. The relaunched task carries a monotonically increasing token (an epoch, a lease number or an attempt number: a number that only grows). The sink, which is the system that receives the writes, refuses writes that carry a stale token.

Example. Attempt 1 holds token 7 and loses its network. The orchestrator launches attempt 2 with token 8. Attempt 2 writes. Later, attempt 1 recovers and tries to write with token 7. The sink has already seen 8, so it rejects 7. Only one copy's output survives.

This is the second thing in this mechanism whose atomicity is borrowed from below rather than owned.

The honest part

Here practice and theory diverge most sharply. Most sinks cannot enforce fencing. An object store accepts any write from any authenticated caller. A JDBC target (a classic database connection) has no concept of your token. A Spark job writing Parquet has no channel for one. So the industry standard is:

at-least-once launch (a task may start more than once, but never zero times),
a timeout tuned by feel,
idempotent tasks to absorb the duplicate, and
hope that the overwrite is atomic enough.

That works most of the time. The residual failure, the torn partition from two concurrent writers, is real, rare and almost impossible to diagnose afterwards.

When someone claims "exactly-once orchestration", this is exactly what they are wrong about. Exactly-once effects need the sink's participation. The orchestrator cannot supply it.

Retry is valid only for transient failures

A deterministic failure (a bad schema, a null-constraint violation, a bug) will fail identically N times. That costs N times the resources and N times the delay before the failure becomes visible. The orchestrator cannot classify failures, because it sees an exit code, not a cause. So the retry count is a bet on the ratio of transient to deterministic failures in your environment. The common default of three is folklore, not a derived number.

The derivable version: set retries high enough to cover the duration distribution of transient failures (a node preemption costs one retry; a cluster autoscaling event perhaps two), and separate the detection of poison pills (inputs that always fail) into a non-retrying path.

You cannot tell dead from slow, so you must fence rather than detect — and since most sinks can't be fenced, at-least-once launch plus idempotency is the real guarantee, not exactly-once.

Case, resolvedConstructed case · Tomás at Marigold Market

730 runs at once, and a torn partition

Rebuilding two years after a bug fix launched 730 runs at once, and a relaunched task wrote alongside its still-running original and tore a partition. Where it started: Safe re-execution →

The 730-run stampede. Tomás sets catchup=False (no automatic runs of past intervals), gives backfills their own pool of 2 slots, and starts the repair by explicit command. His task is a pure function of its key, so a parallel backfill is correct. The cleaning step has no running totals. With 2 parallel runs of 10 minutes each: 730 ÷ 2 × 10 = 3,650 minutes ≈ 61 hours. He accepts that: slow, but production keeps working. He also stamps each output partition with a code_version column, because the fix changes the meaning of one column. Consumers can now see which rows used which logic.

The torn partition. He rewrites the load task as write-audit-publish. Each attempt writes to a staging folder named with its attempt number, asserts row count and unique keys, then swaps one metadata pointer. Two concurrent attempts now write to different staging folders. The pointer swap uses compare-and-swap with the attempt number as token, so a stale attempt loses. The orchestrator still launches at least once, but the visible table cannot be torn.