What role does the Write-Ahead Log (WAL) play in Qdrant, and what failure scenario does it protect against?

Question

Accepted Answer

The WAL is an append-only log of operations on a shard. Every upsert and delete is serialized into the WAL and flushed to disk before, or in parallel with, being applied to the in-memory and on-disk segment state. Its primary job is durability: if the process crashes between applying a write to memory and flushing it into a segment, the write would be lost without a log. On restart, Qdrant replays the WAL from the last known flush point and reconstructs the state that was not yet persisted into segments. Without a WAL, an upsert acknowledged to the client could silently disappear on a crash, which is exactly the failure scenario it protects against. The WAL has a second, equally important role in a distributed cluster: it is the stream that replicates writes from a shard's primary to its replicas. Each replica consumes the WAL and applies the same operations in the same order, which is what keeps replicas consistent without shipping full segments. This is the mechanism behind the consistency levels. When a write is acknowledged depends on write consistency: with write_consistency_factor=1, the primary acks after it has the write locally; with majority or all, the primary waits for that many replicas to confirm they have consumed the WAL entry. The WAL is truncated only after the corresponding state has been flushed into a segment and all replicas have caught up, which is why wal_retain and wal_segments_ahead exist. The unit of truncation is a WAL segment, so the log is a sequence of files that are appended to and then deleted once their contents are durable in segments. Durability: replays unflushed operations after a crash, so acknowledged writes are not lost. Replication: the primary streams WAL entries to replicas, which is how replicas stay in sync. Truncation: WAL files are deleted only after their operations are persisted in segments and consumed by all replicas. Ordering: write_ordering controls whether the WAL is flushed before or after the write is applied, trading latency for durability guarantees. Capacity: wal_capacity_mb sets the size of a single WAL segment, which bounds how much is replayed on restart. The trade-off is durability against write latency. Fsyncing every operation is the safest but slowest; batching amortizes the fsync cost but widens the window in which an acknowledged write could still be lost if the OS buffer is lost. Qdrant exposes this through write_ordering and the wait flag on the write. The common mistake is assuming that a successful upsert response means the data is durable everywhere. It does not - it means the write was accepted, and whether it is durable on one replica, a majority, or all depends on the write consistency factor and the wait flag. A second common mistake is assuming the WAL protects against disk failure. It does not - the WAL lives on the same disk as the data. Protection against disk failure is replication and snapshots, not the WAL. The third mistake is ignoring WAL size. If the WAL is allowed to grow unbounded (e.g. because a replica is far behind), restart times and disk usage can blow up. Version note: the WalConfig fields (wal_capacity_mb, wal_segments_ahead, wal_retain) and the write_ordering enum have changed across releases, and the defaults are not the same in all versions - set them explicitly if you depend on the behavior. from qdrant_client import QdrantClient, models

client = QdrantClient("http://localhost:6333")

client.create_collection(
    collection_name="docs",
    vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
    wal_config=models.WalConfigDiff(
        wal_capacity_mb=32,       # size of a single WAL segment file
        wal_segments_ahead=0,     # pre-allocated WAL segments
        wal_retain=1,             # how many WAL files to keep after flush
    ),
)

# Durability depends on the wait flag and write ordering, not on the WAL alone.
client.upsert(
    collection_name="docs",
    points=[models.PointStruct(id=1, vector=[0.0] * 768)],
    wait=True,   # block until the write is applied and acknowledged per write_consistency
)

# In a replicated cluster, the WAL is also the replication stream from primary
# to replicas. If a replica falls behind, the WAL grows until it catches up. Version-dependent: the WalConfigDiff fields and their defaults have changed. In some releases wal_segments_ahead and wal_retain have different names or semantics, and the write_ordering enum values have been extended. Also, the interaction between the WAL and the optimizer's flush schedule is version-specific - in some versions, segment flush is triggered by WAL size thresholds as well as by the optimizer. If durability is a hard requirement for your workload, benchmark the actual crash-recovery time on your version by killing the process mid-ingest and measuring how long replay takes.

What role does the Write-Ahead Log (WAL) play in Qdrant, and what failure scenario does it protect against?

The WAL provides durability for unflushed writes and feeds replication

What role does the Write-Ahead Log (WAL) play in Qdrant, and what failure scenario does it protect against?

The WAL provides durability for unflushed writes and feeds replication

Scenario Questions

Follow-up Questions