Recall@k, p50/p95/p99 latency, throughput, and indexing time
The core benchmark dimensions are recall, latency, throughput, and indexing time, and they must be measured together because they trade off against each other. Recall@k measures retrieval quality: the fraction of the true top-k results that the system actually returns. Latency measures responsiveness: p50, p95, and p99 for search, because the tail matters more than the median for user experience. Throughput measures capacity: queries per second at a given latency target, and separately writes per second during ingest. Indexing time measures how long it takes to build or rebuild the index, which determines how quickly new data becomes searchable and how long a model upgrade or configuration change takes. A production launch decision needs all four because a system that is fast but has low recall is useless, a system with high recall but a p99 of 500ms is unusable, and a system with good steady-state metrics but a multi-day index build is operationally impractical.
The mechanism behind each metric tells you what to tune. Recall is a function of the HNSW parameters (m, ef_construct), the query-time ef, the quantization configuration, and the embedding model. It is measured against exact ground truth computed by brute force, and it must be measured on a representative query set, not on random vectors. Latency is a function of the same parameters plus the storage layout (in-memory vs on-disk), the number of segments, and the hardware. It must be measured under load, not on a single query, because contention changes the result. Throughput is a function of the parallelism (threads, shards), the query cost, and the hardware; it is the inverse of latency at saturation. Indexing time is a function of the vector count, dimension, m, ef_construct, and the hardware; it determines the cost of a rebuild. In a benchmark, you vary one dimension at a time: fix the corpus and queries, vary ef and measure recall and latency; fix ef, vary m and measure the same; fix both, vary the quantization and measure the same. The result is a set of trade-off curves that let the team choose an operating point.
Recall@k: fraction of true top-k returned, measured against exact ground truth on real queries.
Latency: p50, p95, p99 for search under representative concurrent load.
Throughput: QPS at a latency target, and write throughput during ingest.
Indexing time: time to build the index, which bounds how fast new data becomes searchable.
Memory footprint: RAM used per point, which determines the cost of the deployment.
Storage footprint: disk used per point, which determines the cost of an on-disk deployment.
Cost per query: the total infrastructure cost divided by query volume, for cost-sensitive deployments.
The trade-off is that improving one metric usually degrades another. Higher recall costs latency (higher ef) or memory (higher m). Lower latency costs recall (lower ef) or memory (quantization). Higher throughput costs latency (more concurrency) or hardware. Faster indexing costs recall (lower ef_construct) or hardware. The benchmark's job is to produce the curves so the team can choose an operating point that meets the SLOs at the lowest cost. The common mistake is to benchmark only unfiltered, unfiltered nearest-neighbor queries, which do not represent the production workload if the application uses filters. The second mistake is to benchmark with random query vectors, which have different distributional properties than real queries and produce misleading recall numbers. The third mistake is to measure latency at low concurrency and assume it holds at production load. The fourth mistake is to ignore indexing time until a model upgrade is needed, at which point the rebuild takes days. Version note: the metrics Qdrant exposes and the way recall is affected by quantization and on-disk storage have changed across releases. A benchmark run on one version may not transfer to another, so re-run the benchmark after upgrades.
Version-dependent: the exact behavior of quantization, on-disk storage, and the optimizer affects the benchmark results, and these have changed across Qdrant releases. Re-run the benchmark after upgrading, and always measure recall against exact ground truth on the same version you plan to deploy. The query_points API used here is qdrant-client 1.10+; older clients used search().
0-2 years experience
2-5 years experience
5-8 years experience
8+ years experience
