Isolate by comparing quantization settings against an unquantized baseline
The first step is to establish a baseline: measure recall@k against exact ground truth on the unquantized collection. Without this, you cannot know how much recall was lost or whether the loss is due to quantization at all. The second step is to measure recall on the quantized collection with rescoring and oversampling fully enabled, at the recommended settings for the scheme (e.g. oversampling 3 for binary quantization). This tells you the best-case recall for that quantization configuration. The third step is to measure recall with rescoring off. The difference between rescoring-on and rescoring-off isolates the contribution of rescoring. The fourth step is to sweep oversampling values (1, 2, 3, 5, 10) at rescoring-on, which tells you whether the loss is due to insufficient candidate coverage. If recall recovers with higher oversampling, the problem was the oversampling factor, not the quantization scheme itself. If recall does not recover, the quantization scheme is too aggressive for this data.
The mechanism behind these comparisons is that quantization has two separate effects. The first is on the distance computations during graph traversal: the quantized vectors approximate the true distances, which can cause the traversal to miss the correct neighborhood. The second is on the ranking of the candidate set: even if the correct points are retrieved, the quantized distance may rank them incorrectly. Rescoring addresses the second effect by re-ranking with full-precision vectors, and oversampling addresses the first by retrieving more candidates to compensate for the traversal's imprecision. So if recall is low with rescoring on and high oversampling, the traversal is the problem; if recall is low even with rescoring on and high oversampling, the quantization scheme is too lossy for the data. This decomposition is what lets you pinpoint the cause and choose the right fix. It also matters to compare like with like: if you change ef, m, or the candidate set size at the same time as enabling quantization, you cannot attribute the recall change to quantization.
Baseline: exact ground truth on the unquantized collection, same queries.
Quantized with rescoring on, recommended oversampling: best-case recall for the scheme.
Quantized with rescoring off: isolates the rescoring contribution.
Oversampling sweep: tells you whether the loss is coverage or scheme quality.
Control variables: hold ef, m, and candidate set sizes fixed across all measurements.
Scheme comparison: try scalar vs binary vs product on the same data to see the trade-off curve.
Query distribution: use real queries, not random vectors, because the effect is distribution-dependent.
The trade-off is between accuracy and the cost of the quantization configuration. Higher oversampling recovers recall but increases the rescoring cost, which increases latency. Keeping the quantization scheme and raising oversampling is usually the first fix because it preserves the memory savings. Switching to a less aggressive scheme (binary to scalar, scalar to none) recovers accuracy but gives up memory savings. The common mistake is to measure recall on the quantized collection without rescoring and conclude that quantization is unusable - that is not a fair test because the design assumes rescoring. The second mistake is to change the quantization scheme and the rescoring settings at the same time, which makes it impossible to attribute the recall change. The third mistake is to use random query vectors for the benchmark, which do not reflect the distribution of real queries and can overstate or understate the recall loss. Version note: the supported quantization schemes, the oversampling parameter, and the default rescoring behavior have changed across Qdrant releases. Some versions automatically enable rescoring for binary quantization; others require it to be set explicitly. Verify the defaults on your version before drawing conclusions.
Version-dependent: the quantization search parameters and the default rescoring behavior have changed across Qdrant releases. In some versions, the quantization config includes an always_ram option that affects whether the quantized vectors are used for traversal, and the QuantizationSearchParams shape has evolved. Always set rescore and oversampling explicitly in the benchmark rather than relying on defaults, and benchmark on the same version you plan to deploy.
0-2 years experience
2-5 years experience
5-8 years experience
8+ years experience
