What is the relationship between shard count and query fan-out cost, and why doesn't 'more shards' always mean 'faster'?

More shards means more parallelism, but it also means more fan-out cost. Every query that is not routed to a single shard has to be sent to every shard, each shard runs its local search, and the coordinator collects the results and merges them into a global top-k. The coordinator's work grows linearly with the number of shards: it has to send N requests, receive N responses, and merge N lists. The network cost grows with the number of shards and the size of each response. And the latency is bounded by the slowest shard, not the average, so any skew in shard performance or load is amplified as the shard count grows. At small shard counts, the parallelism dominates and queries get faster. At large shard counts, the coordination overhead dominates and queries get slower, or the latency plateaus. The crossover point depends on the latency of a single shard, the coordinator's processing speed, and the network latency between the coordinator and the shards.

The mechanism behind this is the classic parallel-overhead trade-off. If a query takes T seconds on a single shard and is perfectly parallelized across N shards, the search time drops to T/N, but the coordination adds a fixed cost C plus a per-shard cost F. The total latency is roughly C + NF + T/N. This function has a minimum: as N increases, T/N decreases but NF increases, so there is an optimal N. Beyond that optimal N, adding more shards increases latency. The exact shape depends on the ratio of T to F. For a small collection where T is small, the fixed overhead dominates and sharding does not help. For a large collection where T is large, sharding helps until the coordination cost catches up. There is also a second effect: the coordinator itself becomes a bottleneck at high shard counts, because it has to process N responses and merge N lists, and its CPU or network bandwidth is finite. This is why very high shard counts (hundreds) rarely help a single-query latency, even though they help throughput.

The trade-off is between parallelism and coordination. The common mistakes are: (1) assuming that more shards always means faster queries, when at high N the coordination dominates; (2) not measuring the per-shard latency distribution, so a single hot shard is invisible until the p99 is bad; (3) choosing a shard count that is much higher than needed for the data, which adds overhead without benefit; (4) not using custom sharding for tenant-scoped queries, which would avoid the fan-out entirely; (5) forgetting that the coordinator is a single point for a given request and can become the bottleneck. Version note: the coordinator's implementation and the merge semantics have evolved across Qdrant releases. The prefetch/fusion API changed how multi-stage queries are planned and merged. The behavior at high shard counts may differ. If you are tuning shard count, benchmark the query latency at several shard counts on your version.

Version-dependent: the coordinator's merge semantics, the prefetch/fusion API, and the shard placement logic have changed across Qdrant releases. The crossover point where more shards stops helping is version-specific and depends on the workload. Benchmark on your version rather than relying on a general rule.