How would you retrieve 500 candidates with dense vectors and rerank them using a ColBERT multivector?
Two-stage dense retrieval and multivector reranking
I would make the dense vector search the high-recall candidate generator and set its limit to 500. The second stage would use the ColBERT multivector representation to score those candidates more precisely, returning only the final top-k results.
The reason for this architecture is computational economics. Dense ANN retrieval is comparatively cheap and can search a large collection, while late-interaction scoring is more expensive. Restricting ColBERT scoring to 500 candidates gives the reranker a manageable workload while preserving most of the candidate recall.
The important detail is that the ColBERT representation must already be stored as a compatible multivector in Qdrant, and the query must target that vector configuration. The exact query syntax for multivectors and late interaction is version-sensitive, so I would use the API corresponding to the deployed Qdrant release.
A common mistake is setting the prefetch limit equal to the final top-k. That leaves the reranker too few candidates and can cap recall before reranking even starts.
- 1
Dense ANN is used for high-recall candidate generation
- 2
The reranker operates only on the prefetched candidates
- 3
ColBERT uses a multivector late-interaction representation
- 4
Candidate count should be tuned for recall, latency, and reranking cost
Scenario Questions
0-2 years experience
- 1Why would a retrieval system use a cheap first-stage search before an expensive reranker?
- 2If the relevant document is not among the 500 dense candidates, can the ColBERT reranker recover it? Why not?
2-5 years experience
- 1Your reranker is accurate but too slow when given 2,000 candidates. How would you tune the first-stage candidate count?
- 2Increasing dense candidates from 500 to 1,000 improves recall slightly but doubles reranking latency. How would you evaluate the trade-off?
5-8 years experience
- 1Your ColBERT stage improves precision but creates a p99 latency regression. How would you optimize the two-stage retrieval pipeline?
- 2Dense retrieval has strong recall for English but weak recall for multilingual queries. Would increasing the candidate count solve the problem? How would you investigate?
8+ years experience
- 1You need to serve a multimodal RAG workload with dense retrieval followed by late interaction under a strict latency SLO. How would you architect and tune candidate generation versus reranking?
- 2A new reranker improves NDCG substantially but increases compute cost by 10x. How would you decide where in the retrieval pipeline it belongs?
Follow-up Questions
- Why is 500 a candidate count rather than a fixed best practice?
- What happens if the dense prefetch misses a relevant document that ColBERT would have ranked highly?