Failover: automatic, manual, fenced

Why must the old primary be fenced at the resource?

What it is. Everything above reconstructs after the failure and charges the cost to RTO. A standby already holds the state, so RTO shrinks to detect + promote. You pay continuously instead of at recovery time. A rare, enormous cost is converted into a constant, small one: variance (unpredictability) reduced by paying with volume.

The detection trade-off is about information

Detection is a timeout, and you cannot improve detection speed and detection accuracy at the same time (Failure detection). For failover, the false alarms are spurious failovers, and that is what the arithmetic below has to price.

Automatic or manual is arithmetic, not maturity

Automatic failover pays only if

P(real failure) × downtime saved > P(false alarm) × cost of a spurious failover

So where failover is cheap and lossless (stateless services, read replicas), automation is right. Where it is expensive or lossy (a write primary behind asynchronous replication, where a spurious promotion can lose committed data and cause a split brain, two machines both acting as primary), mature teams deliberately keep a human in the loop.

Fencing is required, because the old primary can never be told

The old primary was silent because it was slow, not dead. A slow node wakes up believing it is still the primary. Nothing the new primary does can make this safe, and no agreement among the live nodes helps, because the relevant node is not taking part. Safety is available only at the shared resource, which must reject a writer holding an older epoch: a fencing token, or STONITH where the resource cannot check one. The pause that wakes a zombie is often a garbage-collection pause, the same "slow, not dead" event that triggered the failover.

The lease sets a floor under RTO

Safe takeover requires waiting out the lease duration plus the maximum clock drift. Shortening the lease to cut RTO increases renewal traffic and raises the chance of losing the lease under load by accident. The knob has a floor, set by the worst pause you tolerate.

Failover converts RTO into RPO

An asynchronous standby is behind by the replication lag. So the data loss at promotion equals the lag at the moment of failure, and lag is highest under exactly the load that caused the failure. The RPO you measure in normal times is not the RPO you get. The lag and the failure are correlated, and the correlation runs against you.

One more cost hides here: a standby that is never exercised is an undeclared dependency with a cold cache, and a fleet sized on the assumption that failover is rare.