Hybrid search, reranking, better chunking, and query rewriting
The symptom - topically related but not answering - is a precision problem, not a recall problem. The retriever is finding chunks in the right neighborhood but the ranking is not precise enough to surface the chunks that specifically answer the question. The most effective first fix is reranking: retrieve a larger candidate set (e.g. 50-100 chunks) with the fast vector search, then rerank them with a cross-encoder or a late-interaction model (ColBERT-style MaxSim). Cross-encoders score the query and chunk jointly, which is far more accurate than the bi-encoder similarity used for retrieval, but they are too slow to run over the whole corpus - hence the retrieve-then-rerank pattern. The second fix is hybrid search: combine dense retrieval with sparse retrieval (BM25 or SPLADE) and fuse the results with RRF. Sparse retrieval catches exact term matches that dense retrieval misses, which is often the difference for factual questions. The third fix is better chunking: if chunks are too large, the embedding averages over multiple topics and the relevant content is diluted; if they are too small, they lack the context needed to answer. The fourth fix is query rewriting: use the LLM to rewrite the user's question into a clearer query before retrieval, or to generate multiple query variants and merge the results.
The mechanism behind each fix addresses a different failure mode. Reranking addresses ranking precision: the bi-encoder similarity is a coarse signal, and the cross-encoder can distinguish between a chunk that mentions the topic and a chunk that answers the question. Hybrid search addresses vocabulary mismatch: dense retrieval can miss chunks that contain the specific term the user asked about, while sparse retrieval can miss paraphrases. Fusion combines both. Chunking addresses context: a chunk must contain enough information to answer, but not so much that its embedding is a blur. Query rewriting addresses query formulation: users often ask vague or multi-part questions that embed poorly, and a rewrite can turn them into a clear retrieval query. The fixes are complementary and can be combined: hybrid retrieval with reranking over well-chunked data is the standard high-quality RAG pipeline. The order to try them depends on which is easiest to implement and measure in your system - reranking is usually the highest-leverage single change, followed by hybrid search and query rewriting.
Reranking: retrieve 50-100 candidates, rerank with a cross-encoder or MaxSim.
Hybrid search: dense + sparse retrieval with RRF fusion.
Better chunking: tune chunk size and overlap, or use semantic chunking.
Query rewriting: LLM rewrites the question into a clearer retrieval query.
Multi-query: generate several query variants and merge results.
Metadata filters: restrict to the right document type, date range, or section.
Embedding model: a stronger model improves both retrieval and reranking.
Evaluation: measure recall@k and MRR/nDCG before and after each change.
The trade-off is latency and complexity against quality. Reranking adds a second stage with its own latency, especially for cross-encoders that run on GPU. Hybrid search requires maintaining a sparse index in addition to the dense index, and the fusion adds a step. Query rewriting adds an LLM call before retrieval, which increases latency and cost. The right combination depends on the quality target and the latency budget. The common mistake is to try to fix precision by increasing the number of retrieved chunks without reranking, which just gives the LLM more noise. The second mistake is to skip the evaluation step and make multiple changes at once, so you cannot tell which helped. The third mistake is to assume the embedding model is the problem when the chunking or the ranking is. The fourth mistake is to ignore sparse retrieval, which is often the difference for factual and entity-heavy queries. Version note: the hybrid search and reranking capabilities in Qdrant - sparse vectors, prefetch, fusion, multivector MaxSim - have evolved across releases. The prefetch/fusion API in qdrant-client 1.10+ lets you express hybrid retrieval and reranking in a single query. If you are on an older version, the pipeline is orchestrated in the application with multiple calls.
Version-dependent: the prefetch/fusion API, sparse vector support, and multivector MaxSim reranking are recent additions to Qdrant and qdrant-client. The exact shape of the nested prefetch and the supported fusion modes (RRF, DBSF) have evolved. If you are on an older version, the hybrid retrieval and reranking are orchestrated in the application layer with multiple calls, which has the same effect but a different code shape and higher round-trip latency. The query_points API is qdrant-client 1.10+; older clients used search() and search_batch().
0-2 years experience
2-5 years experience
5-8 years experience
8+ years experience
