When designing a retrieval system that combines dense, sparse, and multivector reranking at extreme scale, what's the single biggest cost driver you'd optimize first, and why?

The single biggest cost driver is the size of the candidate set that feeds the multivector reranker, because the reranker's cost scales with the number of candidates and each candidate's reranking cost is much higher than the retrieval cost. A MaxSim reranker compares every query token to every document token, so the per-candidate cost is O(q * t), where q is the query length and t is the document length. For a query of 20 tokens and a document of 200 tokens, that is 4000 comparisons per candidate. If the candidate set is 100, that is 400,000 comparisons per query; if it is 1000, it is 4 million. The retrieval stages (dense and sparse) are O(log N) per query and cost far less. So the reranker dominates the total cost, and the candidate set size is the multiplier that determines how much it dominates. Reducing the candidate set from 1000 to 100 cuts the reranker cost by 10x, which is the highest-leverage optimization. The other levers - reducing q (query length) and t (document length) - are less controllable, and the retrieval stages are already cheap. So the first optimization is to tune the prefetch limits to the smallest candidate set that still meets the recall target, and to make the retrieval stages precise enough that a small candidate set is sufficient.

The mechanism that makes the candidate set the dominant cost is the compounding of two costs: the retrieval cost (which produces the candidate set) and the reranking cost (which consumes it). The retrieval cost is bounded and grows slowly with the collection size, because the ANN index gives sub-linear search. The reranking cost grows linearly with the candidate set size and quadratically with the document length. At extreme scale, the retrieval is fast and cheap; the reranker is the bottleneck. The design implication is that the retrieval stages should be tuned to maximize recall at a small candidate set, so that the reranker has fewer candidates to process. This means using a strong dense model, a good sparse model, and fusion (RRF) to combine them, so that the top-100 candidates already contain most of the relevant documents. The reranker then refines the ordering of those 100, which is cheap. If the retrieval is weak and the candidate set must be 1000 to achieve the recall target, the reranker cost is 10x higher. The second design implication is to consider whether the reranker is needed at all for every query: a cheap first-stage ranking might be sufficient for many queries, and the expensive reranker can be reserved for the queries that need it. The third is to reduce the document token count (t) by chunking more aggressively or by pruning tokens that do not contribute to the matching.