How does late-interaction scoring (as used by ColBERT-style models) with MaxSim differ from comparing two single dense vectors?

Single dense vector retrieval compares two vectors - one for the query, one for the document - with a single dot product or cosine similarity. The entire semantic content of the query is compressed into one vector and the entire document into another, and the score is a single scalar. This is a late-binding-free approach: all the matching happens at the embedding level, and the retrieval model has no ability to attend to specific parts of the query or document at scoring time. Late interaction, as used by ColBERT, changes this in two ways. First, both query and document are represented as sets of token-level vectors rather than single vectors. Second, the score is computed by comparing every query token vector to every document token vector and then aggregating - specifically, for each query token, take the maximum similarity over all document tokens, then sum those maxima. This is the MaxSim operation. The result is that the model can match different parts of the query to different parts of the document, capturing term-level and phrase-level alignment that a single-vector score cannot express.

The mechanism difference matters for retrieval quality. A single dense vector has to average over all the concepts in a document, so a query about one specific aspect of a long document can be drowned out by the document's other content. Late interaction preserves the individual token representations, so a query token that matches one specific document token contributes strongly to the score even if the rest of the document is about something else. This is why ColBERT-style models tend to outperform single-vector models on tasks that require fine-grained matching, such as question answering over long documents, code search, and multi-hop retrieval. The cost is computational: a single dense comparison is O(d) where d is the embedding dimension, while MaxSim is O(q * d * t) where q is the number of query tokens and t is the number of document tokens. For a 20-token query and a 200-token document, that is 4000 times more work per comparison - which is exactly why late interaction is used as a reranker over a small candidate set rather than as a first-stage retriever over the whole collection.

The trade-off is quality against cost, and the right choice depends on the retrieval task. For a large-scale first-stage retriever where you need to score millions of documents in milliseconds, single dense vectors are the only practical option. For reranking a top-100 candidate set where quality matters and the candidate set is small, late interaction is often worth the cost. The common mistake is treating late interaction as a drop-in replacement for single-vector retrieval and then being surprised by the latency. It is not a replacement; it is a different stage in the pipeline. The second common mistake is assuming that a better single-vector model can always match ColBERT. On some tasks it can, but the token-level alignment that late interaction provides is a structural advantage, not just a matter of model quality. Version note: MaxSim support in Qdrant is provided through the multivector field and the MAX_SIM comparator, which are relatively recent additions; the exact API and performance characteristics may differ across releases.

Version-dependent: MaxSim in Qdrant is exposed through the multivector field type with multivector_config=MultiVectorConfig(comparator=MultiVectorComparator.MAX_SIM). This API was added in recent releases and has evolved; older versions may not support multivector fields at all, in which case you would implement late interaction in your application layer by fetching token vectors and computing MaxSim yourself.