How would you construct a ground-truth dataset to measure recall accurately when benchmarking ANN parameters?

Ground truth for recall measurement is the exact top-k nearest neighbors for each query, computed by brute force against the full corpus. In Qdrant, this is done by running the same query with exact=True in SearchParams, which forces a full scan and returns the true nearest neighbors. The ground truth must be computed on the same corpus, with the same distance metric, the same vectors, and the same query set that will be used to measure the ANN results. If any of these differ, the recall measurement is meaningless. The output of the ground-truth computation is a mapping from query ID to the set of true top-k point IDs, which you then compare against the ANN results query by query. Recall@k is the average fraction of the true top-k that the ANN result contains, and it must be measured at the exact operating point (ef, m, quantization settings) that you plan to ship.

The most important design decision is the query set. It must be representative of the production query distribution, because recall is distribution-dependent - some queries are easy (the nearest neighbors are well-separated) and some are hard (the nearest neighbors are crowded), and the mix determines the average. Random vectors are not representative because they do not have the same clustering or the same query-to-corpus relationship as real queries. The right approach is to use a sample of real production queries, or a held-out set from the same source as the corpus, or a synthetic set constructed to match the distribution (e.g. paraphrases of corpus documents, or known queries with known relevant documents). The size of the query set matters for the statistical power of the recall estimate: a few hundred queries give a rough estimate, a few thousand give a stable one. The corpus itself should be large enough to be representative but small enough that the brute-force ground truth is feasible - for a 10M-point corpus, a full scan per query is seconds, which is acceptable for a few hundred queries but not for thousands. For very large corpora, you can compute ground truth on a representative subset of the corpus, with the caveat that the recall you measure is relative to that subset, not the full corpus.

The trade-off is between accuracy and cost. Computing exact ground truth on a large corpus with many queries is expensive, so you either reduce the number of queries, reduce the corpus size, or sample both. Each of these reduces the statistical confidence of the recall estimate. The right balance depends on how precise the decision needs to be - for choosing between two configurations that differ by a few points of recall, you need more queries; for a coarse check, a few hundred suffice. The common mistake is to compute ground truth on a different corpus or with a different metric than the one being measured, which produces a recall number that is not meaningful. The second mistake is to use random vectors, which have a different distribution than real queries and produce recall numbers that do not reflect production. The third mistake is to compute ground truth once and reuse it across configuration changes that alter the metric or the corpus. The fourth mistake is to forget to measure recall at the exact operating point - a recall number at ef=128 is not the recall you will see at ef=32. Version note: exact search in Qdrant is controlled by the exact flag in SearchParams, and the query API has changed across releases. Verify the exact-search flag and the query shape for your version before computing ground truth.

Version-dependent: exact search is controlled by SearchParams(exact=True) in recent qdrant-client versions; older clients may have a different flag or require a separate collection with a brute-force configuration. The query_points API is qdrant-client 1.10+. For very large corpora, computing exact ground truth in Qdrant may be slow, and an external implementation (e.g. numpy brute force on the raw vectors) may be faster. The principle is the same: the ground truth must be the exact nearest neighbors under the same metric on the same corpus.