Rebalancing: a controller with a loop delay measured in hours

Why is a rebalancer a slow controller, and why is coupling it to a failure detector dangerous?

Moving a partition consumes the same disks and links that serve the workload. Therefore movement must be throttled. Therefore convergence is slow. Therefore the system spends most of its life in a partially balanced state, and "balanced" is a direction rather than a condition.

The composition hazard

The sharper hazard is the combination of two automatic mechanisms. Automatic failure detection plus automatic rebalancing is a closed loop:

1. A node judged dead triggers re-replication.
2. Re-replication consumes bandwidth.
3. Consumed bandwidth raises latency on healthy nodes.
4. Raised latency produces more false death judgements.
5. Return to step 1.

Every serious system therefore rate-limits the rebalance, or requires a quorum-agreed membership change instead of a single detector's opinion, or demands a human decision. The general theory of such loops, where recovery spends the capacity it needs, belongs to death spirals. What matters here is that the loop is created by the pairing, not by either half, and that the relief mechanism feeds the condition it relieves.

The movement protocol

The movement protocol is forced by the ownership constraint. You cannot move state. You can only copy, catch up, cut over atomically, then retire.

1. Copy: stream a snapshot of the partition to the new node.
2. Catch up: replay the writes that arrived during the copy.
3. Cut over: change ownership in one atomic step (a single designated event).
4. Retire: delete the old copy.

During catch-up the new replica is present, consuming bandwidth and storage, and not yet useful. So the peak footprint of a rebalance exceeds the steady state, which is why clusters run out of space during the operation intended to relieve them. The cutover is the one atomic event, and everything before it is reversible. That is the whole reason for the shape.

What the balancer balances

Placement optimisers balance what they can measure: bytes, partition counts, sometimes leader counts. Workloads skew on what they cannot: request rate per key, CPU per request, working-set residency in memory. So the imbalance you feel is never the imbalance the balancer minimises.

Leadership balance is the most common instance. Leaders take all writes and often all strongly consistent reads. So a perfectly balanced replica distribution can be a badly skewed load distribution. That is why leadership rebalancing exists as a separate mechanism (preferred-leader election, lease rebalancing) acting on the same objects as replica rebalancing.

The objective function is a proxy for the thing you care about, and the residual is systematic rather than random. It correlates with exactly the unusual workloads, which are the ones that hurt. A query optimizer's cost model standing in for runtime, a memory grant sized from a cardinality estimate and a cache hit rate used as the objective when miss costs span six orders of magnitude are the same pattern; here bytes stand in for load.

Compression: rebalancing competes with serving, so it is throttled, so it is slow, so it is a controller too laggy to chase load and too dangerous to couple to a failure detector; and what it balances, bytes, is only a proxy for load.