When the document exceeds the model's context or when retrieval needs fine granularity
Chunking makes sense in two situations. The first is a hard constraint: the embedding model has a maximum context length (e.g. 512 tokens for many sentence-transformer models), and a document longer than that cannot be embedded as a single vector without truncation or a hierarchical model. Truncating loses information; chunking preserves it. The second is a retrieval-quality reason: a single embedding for a long document averages over all its topics, so a query about one specific section may not rank the document highly even though the section is relevant. Chunking produces multiple embeddings per document, each representing a smaller, more focused unit of text, which improves recall for section-level queries. The cost is that chunking multiplies the number of points by the average number of chunks per document, which increases storage, index size, and query fan-out. The decision is therefore a trade-off between retrieval granularity and resource cost, and it depends on the document length distribution, the query granularity, and the model's context limit.
The mechanism that makes chunking effective is that each chunk is embedded in the same vector space as the query, so a query about a specific topic is closer to the chunk that contains that topic than to the document-level average. This is why chunking is standard in RAG systems: the retriever returns chunks, not documents, and the generator uses the chunks as context. The payload design must support tracing chunks back to their source document so that the application can deduplicate results (if a document has multiple matching chunks, show the document once), rank them (by the best chunk score or by an aggregate), and display the source context. The standard payload includes a document_id, a chunk_index, the chunk text (or a reference to it), and any metadata that the document carries (date, category, permissions). If the application needs document-level filters, those fields must be replicated on every chunk, which increases payload size but keeps the filter efficient. If the application needs to preserve ordering, chunk_index enables reconstruction. If the application needs to avoid returning multiple chunks from the same document, the query can be followed by a deduplication step or by a grouping operation.
Model context limit: if the document exceeds it, chunking is required to avoid truncation.
Retrieval granularity: chunking improves recall for section-level queries.
Payload design: document_id, chunk_index, chunk text or reference, and replicated document metadata.
Deduplication: multiple chunks from the same document may match; the application must handle this.
Filter replication: document-level filters (date, category, permissions) must be replicated on every chunk.
Storage cost: chunking multiplies the number of points by the average chunks per document.
Chunk size: a hyperparameter that trades context per chunk against the number of chunks.
The trade-off is between recall and cost. Smaller chunks improve granularity but increase the number of points and the chance that a query matches multiple chunks from the same document; larger chunks reduce cost but may dilute the embedding. The right chunk size depends on the model and the query pattern, and it is worth tuning. The common mistake is to chunk at a fixed size without considering the document structure - chunking mid-sentence or mid-paragraph can hurt retrieval quality. The second mistake is to forget to replicate document-level metadata on every chunk, which makes document-level filters impossible without a join. The third mistake is to not deduplicate results, which can return ten chunks of the same document when the user expected ten documents. The fourth mistake is to chunk a document that is already short enough to embed as a single unit, which adds cost without benefit. Version note: the multivector field and the late-interaction patterns in Qdrant can be used to represent a document as a set of vectors without creating separate points, which is an alternative to chunking for some use cases. The API for multivector fields has evolved across releases.
Version-dependent: the multivector field and its comparator are recent additions and provide an alternative to chunking-as-separate-points for some use cases. The exact API for creating multivector points and querying them has evolved. If you are on an older version, chunking as separate points is the standard approach and is well supported.
0-2 years experience
2-5 years experience
5-8 years experience
8+ years experience
