What is a segment in Qdrant's storage engine, and why does a collection consist of multiple segments rather than one monolithic index?

A segment is the smallest independently indexed unit in Qdrant's storage engine. It holds a set of points, and it owns its own vector storage, payload storage, payload indexes, and HNSW graph. A collection is a logical container composed of one or more segments, plus a write-ahead log and the optimizer state that governs how segments are created, merged, and vacuumed. When you search a collection, the request fans out to every segment, each segment returns its local top-k, and the results are merged into a global top-k. A segment is not a shard - a shard is a distributed unit that can contain many segments, and a segment lives entirely within one shard on one node.

The reason for multiple segments is that an HNSW graph is very expensive to mutate in place. Inserting a point into a large graph means wiring new edges and potentially rewiring existing ones, which is both slow and concurrency-hostile. The LSM-style solution Qdrant uses is to make existing segments immutable and append new points to a small, fresh segment. That new segment is immediately searchable without touching the large graph. Periodically the optimizer merges several small segments into a larger one, building a single consolidated HNSW graph and payload index in the process, then atomically replaces the sources. This gives three things at once: concurrent writes (new data goes to a small mutable segment while readers query the stable ones), background optimization (merging is off the critical write path), and bounded per-write cost (you never rewire the whole graph on an upsert). The cost is fan-out: searching N segments means N graph traversals plus a merge, and each segment has fixed overhead for its indexes and file handles. So there is a sweet spot for segment count - too few and merges are expensive and writes stall, too many and every query pays fan-out overhead.

The trade-off is write throughput and concurrency against read fan-out and merge cost. More, smaller segments make writes fast and merges cheap but increase per-query overhead; fewer, larger segments make queries cheaper but make merges expensive and can stall ingest. The optimizer thresholds (default_segment_number, max_segment_size, indexing_threshold) are how you steer this. The common mistake is confusing segments with shards. Engineers new to Qdrant often assume that increasing shard_number will reduce per-segment cost, which is true at the cluster level, but segments within a shard are still governed by the optimizer and are not something you scale directly. The second common mistake is assuming that a delete immediately shrinks storage - it does not, the point is tombstoned until the segment is vacuumed, which is why deleted_threshold exists. The alternative to the segment model is a single mutable graph index, which some in-memory libraries use. It is simpler for read-only or low-churn data but performs poorly under high write rates, which is exactly why Qdrant did not choose it. Version note: the exact set of optimizer thresholds and their defaults have changed across releases, so read the effective config at runtime rather than hardcoding values from an older doc.

Version-dependent: the segments_count field and the optimizer_config shape have changed across Qdrant releases. The ability to inspect individual segments and their sizes is not exposed in all client versions - some releases only surface the aggregate count. Treat segment count as an emergent property you observe, not a parameter you set, and re-measure after any upgrade that touches the optimizer, because the defaults have shifted more than once.