How does late-interaction scoring (as used by ColBERT-style models) with MaxSim differ from comparing two single dense vectors?

Single dense vector retrieval compares two vectors - one for the query, one for the document - with a single dot product or cosine similarity. The entire semantic content of the query is compressed into one vector and the entire document into another, and the score is a single scalar. This is a late-binding-free approach: all the matching happens at the embedding level, and the retrieval model has no ability to attend to specific parts of the query or document at scoring time. Late interaction, as used by ColBERT, changes this in two ways. First, both query and document are represented as sets of token-level vectors rather than single vectors. Second, the score is computed by comparing every query token vector to every document token vector and then aggregating - specifically, for each query token, take the maximum similarity over all document tokens, then sum those maxima. This is the MaxSim operation. The result is that the model can match different parts of the query to different parts of the document, capturing term-level and phrase-level alignment that a single-vector score cannot express.

The mechanism difference matters for retrieval quality. A single dense vector has to average over all the concepts in a document, so a query about one specific aspect of a long document can be drowned out by the document's other content. Late interaction preserves the individual token representations, so a query token that matches one specific document token contributes strongly to the score even if the rest of the document is about something else. This is why ColBERT-style models tend to outperform single-vector models on tasks that require fine-grained matching, such as question answering over long documents, code search, and multi-hop retrieval. The cost is computational: a single dense comparison is O(d) where d is the embedding dimension, while MaxSim is O(q * d * t) where q is the number of query tokens and t is the number of document tokens. For a 20-token query and a 200-token document, that is 4000 times more work per comparison - which is exactly why late interaction is used as a reranker over a small candidate set rather than as a first-stage retriever over the whole collection.