How would you benchmark whether a proposed quantization configuration is worth the accuracy trade-off for your workload?

A quantization benchmark has three outputs: how much memory it saves, how much latency it changes, and how much recall it costs. The memory and latency parts are easy - compare the collection's reported size and measure p50/p99 latency at a fixed query load. The recall part is the one teams get wrong, because it requires ground truth. The correct method is to run an exact search (brute force, no quantization, no HNSW approximation) over the same queries on the same corpus and treat those results as ground truth. Then run the same queries against the quantized collection and compute recall@k as the fraction of ground-truth top-k that appear in the quantized collection's top-k. That number, together with the latency and memory deltas, is the trade-off. Recall must be measured with the exact configuration you plan to ship, including oversampling and rescoring, because those settings materially change the result.

The most important methodological detail is the query set. Using random vectors or a small synthetic query set will give misleading results because quantization error is distribution-dependent - it depends on how the data is clustered and how close the near neighbors are. The benchmark must use queries drawn from the actual production distribution, ideally a few thousand of them, so that the recall estimate has acceptable variance. If you cannot get production queries, use a held-out set from the same source as your corpus, not random noise. The second detail is to control for confounding variables. If you change quantization and also change ef, you cannot attribute the recall difference to quantization. Hold ef, m, and the candidate set sizes fixed across configurations, and vary only the quantization settings. The third detail is to measure at the operating point you will use - if you plan to raise oversampling to 5 to recover recall, benchmark at oversampling 5, not at the default.