WAL replay from the primary, or snapshot transfer for large gaps
When a node recovers and rejoins the cluster, the cluster compares the state of each shard on that node with the state of the current primary for that shard. If the gap is small - the node was down briefly and the WAL still contains the operations it missed - the node catches up by replaying the WAL entries from where it left off. The primary streams the operations, and the recovering replica applies them in order until it is caught up. If the gap is large - the node was down for a long time, or the WAL has been truncated past the point where the node left off - a full snapshot transfer is used. The primary produces a snapshot of the shard's current state, transfers it to the recovering node, and the node applies it and resumes consuming the WAL. The choice between WAL replay and snapshot transfer is automatic and depends on the size of the gap relative to the available WAL history.
The mechanism relies on two properties of the WAL. First, the WAL is a sequence of operations in order, so replaying it from a known point reconstructs the state deterministically. Second, the WAL is not kept forever - it is truncated once the operations are flushed into segments and all replicas have consumed them. The truncation is what makes WAL replay bounded: if a replica is far behind, the entries it needs may have been deleted. The snapshot mechanism handles this case by transferring the current state directly rather than reconstructing it from history. For the recovering node, the process is: identify which shards it hosts, identify the current primary for each, request synchronization, receive either WAL entries or a snapshot, apply them, and then rejoin the replication stream. During this catch-up, the shard remains available through the primary and any other replicas; the recovering node does not serve traffic for that shard until it is caught up. The cluster's overall availability is therefore not affected by the recovery process, only the redundancy level is reduced until the recovery completes.
Small gap: WAL replay from where the node left off.
Large gap: full snapshot transfer followed by WAL replay of recent entries.
Trigger: the size of the gap relative to the available WAL history.
Truncation: WAL entries are deleted once they are flushed to segments and consumed by all replicas, which bounds replay.
Availability: the shard remains served by the primary and other replicas during the recovery; the recovering node does not serve until caught up.
Redundancy: reduced until the recovery completes, which matters if another failure occurs during recovery.
Monitoring: replication lag and catch-up progress are the key metrics to watch during recovery.
The trade-off is between recovery speed and resource consumption. WAL replay is fast and low-impact when the gap is small. Snapshot transfer is slower and consumes network bandwidth and I/O on both the primary and the recovering node, which can affect query latency during the transfer. For very large shards, the snapshot transfer can take a long time, and during that window the cluster is running with reduced redundancy. The common mistake is to assume that recovery is instant and to plan capacity as if the node is immediately back at full redundancy. The second mistake is to set the WAL retention too aggressively, which forces snapshot transfer even for short outages and slows recovery. The third mistake is to ignore the resource impact of snapshot transfer on the primary - a large snapshot can cause query latency spikes, and you may want to schedule recovery during off-peak hours. The alternative to waiting for automatic recovery is to explicitly re-add the node to the cluster and let the cluster place new shards on it, but this requires the node to be healthy and the cluster to have the capacity to move data. Version note: the exact recovery mechanism (WAL replay versus snapshot, the snapshot transfer protocol, and the behavior of the cluster during recovery) has changed across Qdrant releases. Some versions have more efficient incremental recovery; others rely more heavily on snapshots. Test recovery on your version to understand the time and resource cost.
Version-dependent: the recovery protocol, the snapshot format, and the cluster's behavior during recovery have changed across releases. Some versions support incremental snapshots that reduce transfer size; others do not. The exact API for triggering or monitoring recovery may also differ. If recovery time is important for your SLA, benchmark it on your version with your data sizes rather than assuming a fixed duration.
0-2 years experience
2-5 years experience
5-8 years experience
8+ years experience
