Tail latency usually comes from I/O, background work, or query shape outliers
A healthy p50 with a bad p99 almost always means the median request is served from a fast path - cache-resident data, an indexed vector, a simple filter - and the slow tail is hitting a slow path that only a fraction of requests trigger. The first thing to check is the storage layer. On an on-disk collection, a request whose pages are in the page cache returns in microseconds, while a request whose pages have been evicted blocks on a disk read. That is the classic bimodal distribution: most requests hit cache, a few miss and pay the disk latency. Correlating p99 spikes with page-cache miss rate or with disk I/O metrics is the fastest way to confirm. The second common cause is background optimizer activity: segment merges, vacuuming, and index building consume CPU, memory bandwidth, and I/O, and they can cause p99 spikes that coincide with optimizer runs. The third is query shape: a small fraction of queries may have a highly selective filter, a large offset, a multivector rerank, or a payload retrieval that makes them much more expensive than the median.
The diagnosis method is to segment the traffic and measure latency per segment rather than in aggregate. Break queries down by whether they use a filter, by filter selectivity, by whether they use a multivector field, by offset, and by whether they return vectors. Most systems find that the p99 is concentrated in one or two segments - for example, filtered queries on an unindexed payload field, or queries that return multivectors. If the segments all look similar and the p99 is uniformly bad, the cause is more likely systemic: storage I/O, CPU contention, or a node-level resource issue. On a distributed cluster, also check whether the p99 correlates with a particular shard or replica - a single slow shard can dominate the tail because the coordinator waits for the slowest shard. And check the consistency level: consistency=majority or all can make the tail sensitive to the slowest replica.
Cold page cache / disk reads: bimodal latency where cache misses dominate the tail. Check page-cache hit rate and disk I/O.
Background optimizer: merges, vacuums, and index builds compete with search for CPU and I/O. Correlate p99 spikes with optimizer activity.
Query shape: filtered queries on unindexed fields, large offsets, multivector reranks, and vector retrieval concentrate the tail.
Distributed effects: one slow shard or replica sets the p99 because the coordinator waits for the slowest response.
Consistency level: stronger read consistency makes the tail sensitive to the slowest replica.
Resource contention: other processes on the node, CPU throttling, and memory pressure can all show up as tail latency.
The trade-off in fixing tail latency is usually cost against predictability. Keeping more data in RAM reduces cache-miss tails but costs money; tuning the optimizer to be less aggressive reduces p99 spikes but slows ingest and consolidation; adding payload indexes reduces filtered-query tails but costs memory and slows writes. The choice depends on which segment dominates the tail and what the business values. The common mistake is optimizing the median - raising ef or adding CPU to make p50 faster - when the tail is caused by something the median never touches. The second mistake is assuming the tail is caused by the vector search when it is actually caused by payload retrieval, serialization, or the network. The third mistake is ignoring replica and shard skew: a single hot shard can produce a tail that looks mysterious until you break latency down by shard. Version note: the metrics exposed by Qdrant and the client, and the details of the optimizer's scheduling, have changed across releases - use the metrics available on your version and correlate them with client-side latency rather than relying on a fixed diagnostic checklist.
Version-dependent: the exact metrics Qdrant exposes for optimizer activity, segment state, and per-shard latency have evolved, and the optimizer's scheduling behavior has been tuned over releases. If you are diagnosing tail latency on a specific version, check which metrics are available at runtime (collection info, cluster info, and any server-side metrics endpoints) rather than assuming a particular dashboard.
0-2 years experience
2-5 years experience
5-8 years experience
8+ years experience
