How would you structure payload data to support both 'show me items in category X' and 'show me items NOT in category X' filters efficiently?
One indexed keyword field supports both must and must_not
A single keyword payload field with a keyword index supports both 'in category X' and 'not in category X' without any schema change. The 'in' filter uses must with a FieldCondition matching the category value; the 'not in' filter uses must_not with the same condition. Both filters resolve through the same index. The keyword index maps each distinct category value to a posting list of point IDs, so 'in category X' is a lookup of the posting list for X, and 'not in category X' is the complement - the planner can use the index to identify matching points and exclude them, or to prune segments that contain only matching points. The efficiency of the two filters is comparable because both rely on the same index structure. The only difference is that the complement operation may be slightly less efficient if the matching set is large, because the complement is smaller and the engine may choose to scan rather than complement.
The mechanism that makes this work is that the keyword index is a value-to-points mapping, and the filter semantics (must, should, must_not) are applied on top of it. The index is agnostic to whether the filter is inclusion or exclusion; it just resolves the value to a set of points. For 'not in', the engine can either compute the complement of the matching set (efficient when the matching set is small) or scan the collection and evaluate the predicate per point (efficient when the matching set is large). The planner chooses based on statistics. The practical implication for schema design is that you do not need separate fields for inclusion and exclusion - the same field and index serve both. This is also true for arrays: if the category is a list, a keyword index on the array supports both 'contains X' (must) and 'does not contain X' (must_not). The only design consideration is whether the category values are stable and low enough in cardinality that the index is efficient; a very high-cardinality field (e.g. a unique ID) makes the index large and the complement operation less useful.
- 1
One field, one index: a keyword field with a keyword index supports both inclusion and exclusion.
- 2
must: items where the field matches the value.
- 3
must_not: items where the field does not match the value.
- 4
Array support: a keyword index on an array field supports 'contains' and 'does not contain'.
- 5
Planner choice: the engine picks between complement and scan based on selectivity statistics.
- 6
Cardinality: low-to-medium cardinality is ideal; very high cardinality makes the index large.
- 7
No schema change: both filter types use the same payload structure and index.
The trade-off is that exclusion filters can be less efficient than inclusion filters when the excluded value matches most of the collection. If 'not in category X' matches 90 percent of points, the engine has to scan most of the collection either way, and the index does not help. The common mistake is to design separate fields or indexes for inclusion and exclusion, which is unnecessary and wastes memory. The second mistake is to use a high-cardinality field as a category, which makes the index large and the complement operation useless. The third mistake is to assume that must_not is always faster than must, which is not true - it depends on the selectivity of the underlying condition. The fourth mistake is to forget that must_not combined with other conditions can produce unexpected results if the boolean logic is not carefully constructed - for example, must_not combined with must is an intersection of the positive condition with the complement of the excluded condition, which is usually what you want but is worth testing. Version note: the filter semantics (must, should, must_not) and the keyword index behavior have been stable, but the planner's strategy for exclusion filters and the exact index structure have evolved. Test the filter behavior on your version with a small fixture to confirm the expected results.
Version-dependent: the filter API and the keyword index have been stable across recent releases, but the planner's strategy for exclusion filters and the performance characteristics may vary. The query_points API is qdrant-client 1.10+; older clients used search(). If you are tuning a filter-heavy workload, benchmark both inclusion and exclusion filters on your version with your data to confirm the expected performance.
Scenario Questions
- 1
You need to support both 'in category X' and 'not in category X'. Explain whether you need two fields or one.
- 2
A teammate creates a separate field for exclusion. Explain why that is unnecessary and wasteful.
- 1
Your exclusion filter is slow because it matches most of the collection. Explain the mechanism and propose alternatives.
- 2
You need to combine inclusion and exclusion in a single query. Describe the filter and how you would validate it.
- 1
Design a payload schema for a product catalog with hierarchical categories and support for inclusion and exclusion filters at any level.
- 2
You have a filter that excludes items based on a permission field. Describe how you would structure the schema and the filter to keep it efficient.
- 1
Derive the break-even point where an exclusion filter becomes more efficient with a scan than with the index complement, as a function of selectivity and collection size.
- 2
You are designing a query planner for a search system with complex boolean filters. Describe the cost model and how you would choose between index-based and scan-based execution for each clause.
Follow-up Questions
- ❓How would you handle a category hierarchy where items can belong to multiple levels, and how would inclusion and exclusion interact?
- ❓If exclusion filters are slow because the excluded value matches most of the collection, what alternatives do you have?