Plan a partition move
Moving data between nodes consumes the same disks and links that serve the workload, so it must be throttled, so it is slow: the cluster spends most of its life partially balanced. And the hard part is not the bytes. It is the partition map, a piece of state that every client and router caches, with no way to force a refresh. How do you move ownership without an outage?
You will end with a move plan with its duration, its peak space, its one atomic step, and what keeps clients and failure detection from turning it into an incident.
Work through these in order.
Work through it
Move logical shards; do not change their count
Map keys to a large, fixed number of logical shards, then logical shards to nodes. A move is then a pure assignment change: the key-to-shard mapping never changes, only placement does. The logical count is the one number you cannot change later, so it was meant to be chosen generously at the start.
Size the move by its throttle
10 TB at a 100 MB/s throttle takes about 28 hours. That is the loop delay of rebalancing: anything triggered by an instantaneous load metric chases a signal that has already changed. Trigger on sustained, damped signals, with different thresholds for starting and stopping, or decide by hand.
Check free space for the peak, not the steady state
During catch-up the new copy is present, using bandwidth and storage, and not yet serving. The peak footprint of a move exceeds the steady state, which is why clusters run out of space during the operation meant to relieve them.
Copy, catch up, cut over, retire
You cannot move state, only copy it: stream a snapshot to the new node, replay the writes that arrived during the copy, change ownership in one atomic step, then delete the old copy. Everything before the cutover is reversible.
Version the map, and let the owner correct stale clients
Clients will act on a stale map. Tag requests with the map's epoch; the destination rejects an old epoch and returns the new mapping, so the error path is also the refresh path. Expect the least active client to discover the change at the worst moment.
Keep the move away from automatic failure detection
Automatic failure detection plus automatic re-replication is a closed loop: a node judged dead triggers movement, movement consumes bandwidth, bandwidth raises latency on healthy nodes, and latency produces more false deaths. Rate-limit the movement, require a quorum-agreed membership change rather than one detector's opinion, or demand a human decision.
Balance the load, not only the bytes
Balancers minimize what they can measure, bytes and partition counts; workloads skew on request rate, CPU per request and leadership. Leaders take all writes, so a perfectly balanced replica layout can carry a badly skewed load: balance leadership as a separate step.