How would you reduce query latency for a collection that must remain on-disk due to its size, without moving the whole collection into RAM?

When a collection has to stay on disk, latency is dominated by how much data each query has to touch and how much of that data is resident in the page cache. The strategy has three parts: shrink the data the traversal touches, reduce the number of traversal steps, and avoid touching data the query does not need. The most effective single change is quantization with rescoring. Quantized vectors are far smaller, so the working set fits in cache more easily, and the traversal distance computations use the small quantized vectors while only a small candidate set is rescored against the full-precision vectors on disk. Binary quantization with oversampling and rescoring is often the best fit for on-disk collections because it shrinks the traversal data by 32x and the rescoring step touches only a bounded number of raw vectors.

The second lever is inline storage of the quantized vectors in the graph nodes. On an on-disk collection without inline storage, traversing the graph requires one read for the node's edges and a second read for the vector data of each candidate. Inlining the quantized vector into the node turns that into a single read, which roughly halves the I/O per traversal step. This matters most when the collection is I/O-bound, which is exactly the case here. The third lever is reducing the traversal length: a lower ef reduces the number of nodes visited, which reduces I/O at the cost of recall, and a moderate m reduces the number of edges stored per node. The fourth is reducing the candidate set before it reaches the expensive stage - use payload indexes to prune segments and use prefetch to reduce the number of points that need full-precision rescoring. The fifth is exploiting locality: if part of the collection is hot, ensure that portion stays in the page cache by keeping it in RAM or by using a hot/cold tiering approach, so that most queries never touch cold disk pages.

The trade-off is recall and complexity against latency. Every lever that reduces latency either loses some recall (quantization, lower ef, lower m) or adds operational complexity (tiering, index tuning). The right combination depends on how much recall you can give up and how much engineering you are willing to invest. The common mistake is trying to fix on-disk latency purely by tuning ef, which only helps if the traversal is the bottleneck; if the bottleneck is payload retrieval or segment fan-out, ef changes nothing. The second mistake is enabling quantization without rescoring on a collection where recall matters - the latency improvement is real but the accuracy loss may be unacceptable. The third mistake is ignoring the page cache when sizing the machine: a collection that could fit its working set in 32 GB of RAM but is deployed on a 16 GB node will thrash regardless of how well it is configured. Version note: on-disk HNSW, inline storage, and the quantization options have all changed across releases - the available levers differ between versions, so verify which ones your deployment supports before designing the plan.

Version-dependent: on-disk HNSW, inline storage, and the supported quantization schemes have changed across releases, as have the defaults for the optimizer thresholds that decide when a segment moves to disk. The latency improvement from inline storage in particular is version-specific because it depends on the storage layout the current version produces. Benchmark the configuration on your version with your data before committing to it, and re-benchmark after upgrades.