How would you reduce query latency for a collection that must remain on-disk due to its size, without moving the whole collection into RAM?

Question

Accepted Answer

When a collection has to stay on disk, latency is dominated by how much data each query has to touch and how much of that data is resident in the page cache. The strategy has three parts: shrink the data the traversal touches, reduce the number of traversal steps, and avoid touching data the query does not need. The most effective single change is quantization with rescoring. Quantized vectors are far smaller, so the working set fits in cache more easily, and the traversal distance computations use the small quantized vectors while only a small candidate set is rescored against the full-precision vectors on disk. Binary quantization with oversampling and rescoring is often the best fit for on-disk collections because it shrinks the traversal data by 32x and the rescoring step touches only a bounded number of raw vectors. The second lever is inline storage of the quantized vectors in the graph nodes. On an on-disk collection without inline storage, traversing the graph requires one read for the node's edges and a second read for the vector data of each candidate. Inlining the quantized vector into the node turns that into a single read, which roughly halves the I/O per traversal step. This matters most when the collection is I/O-bound, which is exactly the case here. The third lever is reducing the traversal length: a lower ef reduces the number of nodes visited, which reduces I/O at the cost of recall, and a moderate m reduces the number of edges stored per node. The fourth is reducing the candidate set before it reaches the expensive stage - use payload indexes to prune segments and use prefetch to reduce the number of points that need full-precision rescoring. The fifth is exploiting locality: if part of the collection is hot, ensure that portion stays in the page cache by keeping it in RAM or by using a hot/cold tiering approach, so that most queries never touch cold disk pages. Quantize with rescoring: binary quantization (32x) or scalar (4x) keeps the traversal working set small. Inline storage: co-locate quantized vectors with graph nodes to halve the I/O per traversal step. Lower ef and moderate m: fewer nodes visited per query, at the cost of recall. Payload indexes: prune segments and reduce the candidate set before vector search. Prefetch and rerank: use a cheap first stage to produce a small candidate set, then rescore exactly. Hot/cold tiering: keep the hot portion of the collection in RAM or in the page cache and let cold data stay on disk. Storage: NVMe rather than spinning disk makes on-disk latency acceptable for interactive workloads. The trade-off is recall and complexity against latency. Every lever that reduces latency either loses some recall (quantization, lower ef, lower m) or adds operational complexity (tiering, index tuning). The right combination depends on how much recall you can give up and how much engineering you are willing to invest. The common mistake is trying to fix on-disk latency purely by tuning ef, which only helps if the traversal is the bottleneck; if the bottleneck is payload retrieval or segment fan-out, ef changes nothing. The second mistake is enabling quantization without rescoring on a collection where recall matters - the latency improvement is real but the accuracy loss may be unacceptable. The third mistake is ignoring the page cache when sizing the machine: a collection that could fit its working set in 32 GB of RAM but is deployed on a 16 GB node will thrash regardless of how well it is configured. Version note: on-disk HNSW, inline storage, and the quantization options have all changed across releases - the available levers differ between versions, so verify which ones your deployment supports before designing the plan. from qdrant_client import QdrantClient, models

client = QdrantClient('http://localhost:6333')

# On-disk collection optimized for latency without moving everything to RAM:
# - binary quantization with quantized vectors resident in RAM for traversal
# - raw vectors on disk, only touched during rescoring
# - graph on disk with inline quantized vectors
client.create_collection(
    collection_name='docs_disk',
    vectors_config=models.VectorParams(
        size=1024,
        distance=models.Distance.COSINE,
        on_disk=True,                        # raw vectors on disk
    ),
    hnsw_config=models.HnswConfigDiff(
        m=16,
        ef_construct=100,
        on_disk=True,                        # graph on disk
    ),
    quantization_config=models.BinaryQuantization(
        binary=models.BinaryQuantizationConfig(always_ram=True),
    ),
)

# Query: oversample and rescore so accuracy is recovered from the small candidate set.
hits = client.query_points(
    collection_name='docs_disk',
    query=query_vector,
    limit=10,
    search_params=models.SearchParams(
        hnsw_ef=128,
        quantization=models.QuantizationSearchParams(rescore=True, oversampling=3.0),
    ),
).points Version-dependent: on-disk HNSW, inline storage, an

How would you reduce query latency for a collection that must remain on-disk due to its size, without moving the whole collection into RAM?

Shrink the working set, shorten the traversal, and prune with indexes

How would you reduce query latency for a collection that must remain on-disk due to its size, without moving the whole collection into RAM?

Shrink the working set, shorten the traversal, and prune with indexes

Scenario Questions

Follow-up Questions