What role does caching play in a Qdrant-backed search system, and at what layers would you introduce it?

Caching plays three distinct roles, at three layers. At the application layer, an embedding cache stores the vectors produced by the embedding model for repeated inputs. This is the highest-value cache because embedding generation is often the most expensive part of the query path - it is a GPU or API call that takes tens to hundreds of milliseconds - and queries repeat frequently in many applications. A cache hit avoids the embedding call entirely and reduces the query to a vector search. At the query-result layer, a result cache stores the final top-k for repeated queries. This is valuable when the same query is issued many times and the underlying data has not changed, which happens for popular searches and for dashboard queries. The cache key must include the query, the filters, and any parameters that affect the result, and the cache must be invalidated when the underlying data changes. At the OS level, the page cache holds the on-disk data that has been recently accessed, which is automatic and does not require application code. For an on-disk collection, the page cache is the most important cache because it determines whether a query hits RAM or disk.

The mechanism that makes each cache effective is the locality of the workload. Embedding caches exploit the fact that a small fraction of queries account for a large fraction of traffic (the Zipf distribution of search queries). Result caches exploit the same distribution, plus the fact that the underlying data changes slowly relative to the query rate. The OS page cache exploits the fact that the working set of a collection is much smaller than the total data: the hot portion of the HNSW graph and the hot vectors are accessed far more often than the cold portion. Each cache has a different invalidation problem: the embedding cache is invalidated when the embedding model changes, the result cache is invalidated when the underlying data changes, and the page cache is managed by the OS with no application control. The choice of cache depends on the workload and the cost of staleness. In a write-heavy system, the result cache is less useful because it goes stale quickly; in a read-heavy system, it is very effective.

The trade-off is between staleness and latency. A cache that returns stale results is fast but may be wrong; a cache that is invalidated aggressively is correct but has a lower hit rate. The right balance depends on the application: a product search can tolerate a few minutes of staleness, a compliance search cannot. The common mistakes are: (1) caching results without including the filters in the cache key, so a query with a filter returns the cached result of an unfiltered query; (2) not invalidating the result cache when data changes, so stale results persist; (3) not caching embeddings, which is often the biggest single win; (4) assuming that the OS page cache is not a cache and ignoring it when sizing RAM; (5) not measuring the hit rate, so the cache's value is unknown. Version note: the caching layers are application-level and OS-level, not Qdrant-specific. Qdrant does not provide a built-in result cache; the application must implement one. The page cache is managed by the OS and its behavior depends on the host configuration and the workload.

Version-dependent: the caching layers are application-level and OS-level. Qdrant does not provide a built-in result cache, and the page cache behavior is host-specific. The on_disk flags and the optimizer thresholds that affect what stays in RAM have evolved across releases.