When designing a retrieval system that combines dense, sparse, and multivector reranking at extreme scale, what's the single biggest cost driver you'd optimize first, and why?

The single biggest cost driver is the size of the candidate set that feeds the multivector reranker, because the reranker's cost scales with the number of candidates and each candidate's reranking cost is much higher than the retrieval cost. A MaxSim reranker compares every query token to every document token, so the per-candidate cost is O(q * t), where q is the query length and t is the document length. For a query of 20 tokens and a document of 200 tokens, that is 4000 comparisons per candidate. If the candidate set is 100, that is 400,000 comparisons per query; if it is 1000, it is 4 million. The retrieval stages (dense and sparse) are O(log N) per query and cost far less. So the reranker dominates the total cost, and the candidate set size is the multiplier that determines how much it dominates. Reducing the candidate set from 1000 to 100 cuts the reranker cost by 10x, which is the highest-leverage optimization. The other levers - reducing q (query length) and t (document length) - are less controllable, and the retrieval stages are already cheap. So the first optimization is to tune the prefetch limits to the smallest candidate set that still meets the recall target, and to make the retrieval stages precise enough that a small candidate set is sufficient.

The mechanism that makes the candidate set the dominant cost is the compounding of two costs: the retrieval cost (which produces the candidate set) and the reranking cost (which consumes it). The retrieval cost is bounded and grows slowly with the collection size, because the ANN index gives sub-linear search. The reranking cost grows linearly with the candidate set size and quadratically with the document length. At extreme scale, the retrieval is fast and cheap; the reranker is the bottleneck. The design implication is that the retrieval stages should be tuned to maximize recall at a small candidate set, so that the reranker has fewer candidates to process. This means using a strong dense model, a good sparse model, and fusion (RRF) to combine them, so that the top-100 candidates already contain most of the relevant documents. The reranker then refines the ordering of those 100, which is cheap. If the retrieval is weak and the candidate set must be 1000 to achieve the recall target, the reranker cost is 10x higher. The second design implication is to consider whether the reranker is needed at all for every query: a cheap first-stage ranking might be sufficient for many queries, and the expensive reranker can be reserved for the queries that need it. The third is to reduce the document token count (t) by chunking more aggressively or by pruning tokens that do not contribute to the matching.

The trade-off is between recall and reranker cost. A larger candidate set improves recall but increases cost linearly. A smaller candidate set is cheaper but may miss relevant documents. The right balance depends on the recall target and the retrieval quality: with a strong retrieval stage, a small candidate set achieves the recall target; with a weak one, a larger candidate set is needed. The common mistakes are: (1) setting the candidate set size based on intuition rather than measuring recall; (2) over-investing in the reranker while under-investing in the retrieval, so the candidate set must be large; (3) applying the reranker to every query when a cheaper first-stage ranking would suffice for many; (4) not reducing the document token count, so the per-candidate cost is high; (5) not measuring the cost breakdown, so the dominant cost is unknown. Version note: the prefetch/fusion API and the multivector MaxSim reranking have evolved across Qdrant releases. The exact cost of the reranker depends on the version's implementation. Benchmark on your version with your data.

Version-dependent: the prefetch/fusion API and multivector reranking have evolved across Qdrant releases. The exact cost of the reranker depends on the version's implementation. Benchmark on your version with your data.