What role does caching play in a Qdrant-backed search system, and at what layers would you introduce it?

Question

Accepted Answer

Caching plays three distinct roles, at three layers. At the application layer, an embedding cache stores the vectors produced by the embedding model for repeated inputs. This is the highest-value cache because embedding generation is often the most expensive part of the query path - it is a GPU or API call that takes tens to hundreds of milliseconds - and queries repeat frequently in many applications. A cache hit avoids the embedding call entirely and reduces the query to a vector search. At the query-result layer, a result cache stores the final top-k for repeated queries. This is valuable when the same query is issued many times and the underlying data has not changed, which happens for popular searches and for dashboard queries. The cache key must include the query, the filters, and any parameters that affect the result, and the cache must be invalidated when the underlying data changes. At the OS level, the page cache holds the on-disk data that has been recently accessed, which is automatic and does not require application code. For an on-disk collection, the page cache is the most important cache because it determines whether a query hits RAM or disk. The mechanism that makes each cache effective is the locality of the workload. Embedding caches exploit the fact that a small fraction of queries account for a large fraction of traffic (the Zipf distribution of search queries). Result caches exploit the same distribution, plus the fact that the underlying data changes slowly relative to the query rate. The OS page cache exploits the fact that the working set of a collection is much smaller than the total data: the hot portion of the HNSW graph and the hot vectors are accessed far more often than the cold portion. Each cache has a different invalidation problem: the embedding cache is invalidated when the embedding model changes, the result cache is invalidated when the underlying data changes, and the page cache is managed by the OS with no application control. The choice of cache depends on the workload and the cost of staleness. In a write-heavy system, the result cache is less useful because it goes stale quickly; in a read-heavy system, it is very effective. Embedding cache: caches the vectors from the embedding model; avoids the most expensive part of the query path. Result cache: caches the final top-k for repeated queries; must include filters and parameters in the key. Page cache: OS-level, automatic, holds recently accessed on-disk data; critical for on-disk collections. Locality: search queries follow a Zipf distribution, so a small cache covers a large fraction of traffic. Invalidation: embedding cache on model change, result cache on data change, page cache managed by the OS. Staleness: result cache can return stale results; the acceptable staleness depends on the application. Cache key: must include the query, the filters, the limit, and any other parameter that affects the result. Metrics: hit rate, latency of hits vs misses, and the cost saved by each cache. The trade-off is between staleness and latency. A cache that returns stale results is fast but may be wrong; a cache that is invalidated aggressively is correct but has a lower hit rate. The right balance depends on the application: a product search can tolerate a few minutes of staleness, a compliance search cannot. The common mistakes are: (1) caching results without including the filters in the cache key, so a query with a filter returns the cached result of an unfiltered query; (2) not invalidating the result cache when data changes, so stale results persist; (3) not caching embeddings, which is often the biggest single win; (4) assuming that the OS page cache is not a cache and ignoring it when sizing RAM; (5) not measuring the hit rate, so the cache's value is unknown. Version note: the caching layers are application-level and OS-level, not Qdrant-specific. Qdrant does not provide a built-in result cache; the application must implement one. The page cache is managed by the OS and its behavior depends on the host configuration and the workload. from qdrant_client import QdrantClient, models
from functools import lru_cache
import hashlib
import json

client = QdrantClient('http://localhost:6333')

@lru_cache(maxsize=100_000)
def embed_cached(text: str):
    return tuple(embed(text))

def result_cache_key(query_vector, filters, limit):
    payload = json.dumps({'q': list(query_vector), 'f': filters, 'l': limit}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()

result_cache = {}

def search(query_text, filters=None, limit=10):
    qv = embed_cached(query_text)
    key = result_cache_key(qv, filters, limit)
    if key in result_cache:
        return result_cache[key]
    hits = client.query_points(
        collection_name='documents',
        query=list(qv),
        limit=limit,
        query_filter=filters,
    ).points
    result_cache[key] = hits
    return hits

def invalidate_result_cache():
    result_c

What role does caching play in a Qdrant-backed search system, and at what layers would you introduce it?

Embedding cache, query-result cache, and the OS page cache at three different layers

Scenario Questions

Follow-up Questions