What operational steps are involved in adding a new node to an existing Qdrant cluster and rebalancing shards onto it?

Adding a node to an existing Qdrant cluster involves a sequence of steps: provision the node with the same configuration and version as the existing cluster, join it to the cluster's Raft group, wait for the cluster to recognize it, then trigger the placement or rebalancing logic to move some shard replicas onto the new node, and verify the transfer. The exact mechanism depends on the version: in some versions, the cluster automatically distributes shards when a new node joins; in others, the operator must trigger a rebalance. The transfer itself is a data migration: the new node receives a snapshot or a stream of the shard's data from a peer, applies it, and then joins the replication stream. The shard's primary continues to serve traffic during the transfer, so there is no downtime for that shard, but the transfer consumes network bandwidth and disk I/O on both the source and the destination node, which can affect query latency. The process must be done one shard at a time, or in small batches, to avoid saturating the network and causing latency spikes.

The mechanism that makes this safe is that shard replicas are independent: moving one replica of a shard does not affect the shard's availability, because the primary and the other replicas continue to serve. The cluster's metadata (maintained through Raft) tracks the placement, and the rebalance updates the placement atomically. The new node's replica is only promoted to primary if the primary fails, so the rebalance does not change the write path. The transfer is bounded by the size of the shard and the network bandwidth between the source and the destination. For a large shard, this can take minutes to hours. During the transfer, the source node's query latency may increase because of the I/O and network load, so the transfer should be scheduled during a low-traffic period if possible. After the transfer, the new node's replica is fully in sync and can serve reads, which increases the cluster's capacity. The rebalance is complete when the cluster's placement is balanced, which may require moving multiple replicas. The whole process should be monitored for progress, latency impact, and errors.