Deterministic chunk IDs plus document, section, page, and offset metadata
The design has two parts: a chunk ID scheme that is deterministic and traceable, and a payload schema that carries the provenance. The chunk ID should be derived from the source document and the chunk's position within it, e.g. '{document_id}::{chunk_index}' or a hash of the document ID plus the chunk's character offset. A deterministic ID makes upserts idempotent, so re-processing a document overwrites the existing chunks rather than duplicating them. The payload should carry at minimum: document_id, chunk_index, section (heading or logical section), page (if the source has pagination), start_offset and end_offset (character or token offsets within the source), and the source URI or an object-storage key. It should also carry the chunk text itself, because the generation stage needs the text to build the prompt and does not want to fetch it from a separate store. Document-level metadata (title, publication date, category, access permissions, tenant) should be replicated on every chunk so that filters on document-level attributes work without a join.
The mechanism that makes this work is that the payload is the only information the retrieval stage has about the chunk's origin. When the LLM cites a source, the application needs to map the retrieved chunk back to a human-readable location in the original document. The section and page fields let the application say 'see section 3.2, page 12', which is far more useful than a raw chunk index. The offset fields let the application highlight the exact span of text in a viewer. The document_id lets the application deduplicate - if three chunks from the same document are retrieved, the application can show the document once with the relevant sections highlighted. The replicated document metadata means the retrieval filter can be applied at the chunk level without a separate lookup: a query that filters by tenant or publication date works directly on the chunk collection. This is why the payload schema must be designed before ingestion starts - adding fields later requires re-ingesting the corpus.
Chunk ID: deterministic, derived from document ID and chunk index or offset.
document_id: stable identifier for the source document.
chunk_index: position within the document, used for ordering and reconstruction.
section: heading or logical section name for human-readable citation.
page: page number if the source has pagination (PDF, printed docs).
start_offset / end_offset: character or token offsets within the source for highlighting.
chunk_text: the chunk's text, so the generation stage does not need a separate lookup.
source_uri: a link to the original document for the user to open.
document-level metadata replicated: title, date, category, permissions, tenant.
Index the filterable fields: document_id, date, category, permissions, tenant.
The trade-off is between payload richness and payload size. Every field in the payload adds storage and, if indexed, memory and write cost. Replicating document-level metadata on every chunk multiplies the metadata storage by the average number of chunks per document, which for a corpus with long documents can be significant. The chunk text itself is the largest field; storing it in the payload is convenient but increases payload size. An alternative is to store the chunk text in a separate store and put a reference in the payload, but that adds a lookup step and a dependency during generation. In practice, storing the chunk text in the payload is the right default because the generation stage needs it immediately, and the alternatives add complexity without a clear benefit. The common mistake is to not store enough metadata to support citation, so the answers cannot be verified. The second mistake is to use non-deterministic chunk IDs, so re-processing creates duplicates. The third mistake is to not replicate document-level metadata on chunks, so filters on document attributes do not work. The fourth mistake is to store the chunk text only as a reference, adding a lookup that the generation stage has to perform under latency pressure. Version note: the payload index types (keyword, datetime, integer, etc.) and the is_tenant flag have evolved across Qdrant releases; design the schema with the version's capabilities in mind, and consider which fields need indexes based on the filter patterns you expect.
Version-dependent: the payload index types and the API for creating them have changed across Qdrant releases. The is_tenant flag and the on_disk option for indexes are recent additions. The query_points and upsert API shape is qdrant-client 1.10+. If you are on an older version, verify which payload schema types are available before designing the schema.
0-2 years experience
2-5 years experience
5-8 years experience
8+ years experience
