How does a payload filter interact with segment selection during query execution - does Qdrant always scan every segment?

Qdrant does not always scan every segment. A payload filter is planned against the payload indexes and segment-level statistics to determine which segments can possibly contain matching points, and segments that cannot are skipped entirely. The simplest form of pruning is a segment that contains no points at all for the filtered value - for example, if a segment has a payload index on tenant_id and the filter is tenant_id = "tenant_a", a segment whose index shows no tenant_a points is skipped. More sophisticated pruning uses range statistics: if a filter is created_at >= "2024-01-01" and a segment's index records a maximum created_at before that date, the segment can be skipped. This pruning happens before any vector search, so it reduces the number of HNSW traversals the query has to perform, which is a direct saving on both latency and CPU.

The mechanism depends on the type of payload index. A keyword index maps values to posting lists of point IDs; a filter on a keyword can be resolved to a set of matching points, and segments whose posting lists are empty for the filtered value are skipped. An integer or datetime index supports range queries and can be pruned using min/max statistics at the segment level. A full-text index supports term-based pruning. Without an index, the filter has to be evaluated by reading each point's payload, which means no segment-level pruning is possible and every segment in the shard has to be traversed with the filter checked per candidate - this is the expensive case. Even with an index, the planner has to choose between using the index to build a candidate bitmap and using it only to prune segments; for very selective filters the index-first approach wins, while for less selective filters the filter-aware HNSW traversal may be cheaper. This is a cost-based decision and the planner's behavior has evolved across versions.

The trade-off is index maintenance and memory against query pruning. Payload indexes cost memory and slow down ingest slightly because each upsert has to update the index, but they enable segment pruning and filter-aware traversal, which can be a large latency win for selective filters. The common mistake is not creating a payload index on a field that is filtered frequently, which forces the engine to check every candidate's payload and prevents segment pruning. The second mistake is creating an index on a field with very low cardinality (e.g. a boolean) and expecting a large speedup - the index helps, but the pruning benefit is limited because most segments will contain both values. The third mistake is assuming that a filter always reduces the search cost. A filter that matches almost everything does not reduce the number of candidates the traversal has to consider, and can add overhead because the filter check runs per candidate. Version note: the set of supported payload index types and the planner's pruning logic have changed across releases; the exact behavior under a combined filter (multiple conditions with must/should/must_not) is version-specific and worth benchmarking on your own data.

Version-dependent: payload index types, the query planner's pruning behavior, and the ability to combine index-based and scan-based execution have evolved across Qdrant releases. The introduction of the prefetch/fusion API also changed how filters interact with multi-stage queries - in a prefetch, the filter is applied at the stage where it is specified, and the planner's behavior at each stage is version-specific. If you are tuning filtered search performance, create the indexes you expect to need, benchmark with your actual filter selectivity, and verify the planner's behavior on your version rather than assuming a particular pruning strategy.