What is the relationship between shard count and query fan-out cost, and why doesn't 'more shards' always mean 'faster'?

More shards means more parallelism, but it also means more fan-out cost. Every query that is not routed to a single shard has to be sent to every shard, each shard runs its local search, and the coordinator collects the results and merges them into a global top-k. The coordinator's work grows linearly with the number of shards: it has to send N requests, receive N responses, and merge N lists. The network cost grows with the number of shards and the size of each response. And the latency is bounded by the slowest shard, not the average, so any skew in shard performance or load is amplified as the shard count grows. At small shard counts, the parallelism dominates and queries get faster. At large shard counts, the coordination overhead dominates and queries get slower, or the latency plateaus. The crossover point depends on the latency of a single shard, the coordinator's processing speed, and the network latency between the coordinator and the shards.

The mechanism behind this is the classic parallel-overhead trade-off. If a query takes T seconds on a single shard and is perfectly parallelized across N shards, the search time drops to T/N, but the coordination adds a fixed cost C plus a per-shard cost F. The total latency is roughly C + NF + T/N. This function has a minimum: as N increases, T/N decreases but NF increases, so there is an optimal N. Beyond that optimal N, adding more shards increases latency. The exact shape depends on the ratio of T to F. For a small collection where T is small, the fixed overhead dominates and sharding does not help. For a large collection where T is large, sharding helps until the coordination cost catches up. There is also a second effect: the coordinator itself becomes a bottleneck at high shard counts, because it has to process N responses and merge N lists, and its CPU or network bandwidth is finite. This is why very high shard counts (hundreds) rarely help a single-query latency, even though they help throughput.