What is the effect of increasing the number of search threads/parallelism on a single node with limited CPU cores?

Question

Accepted Answer

Increasing search threads on a single node has the classic parallelism curve: initially throughput scales roughly linearly as more cores are used, then it plateaus as the cores are saturated, and then it degrades as oversubscription causes context switching, cache thrashing, and memory-bandwidth contention. For a CPU-bound search workload, the sweet spot is at or slightly below the number of physical cores, and going beyond that is rarely beneficial. The specific shape of the curve depends on whether the workload is compute-bound or memory-bound. Distance computations are compute-heavy, so more threads help until the cores are busy. But the HNSW traversal involves random memory access, which is memory-bandwidth-bound, so beyond a certain point additional threads contend for the same memory bus and the marginal throughput is near zero or negative. On on-disk collections, the bottleneck may be I/O rather than CPU, in which case more threads can increase I/O queue depth and help - up to the point where the disk's queue is saturated. There is an interaction with segments and shards that is easy to miss. A single search over a collection is parallelized across segments and across shards, so the amount of work available to parallelize is bounded by the number of segments and the number of shards on the node. If you set the thread count higher than the available parallelism, the extra threads have nothing to do and you get no benefit - or you get coordination overhead. This is why the right thread count depends on the collection layout, not just on the CPU count. On a node with a few large segments, there is less parallelism to exploit than on a node with many small segments. Similarly, a collection with 4 shards on a node can use at most 4-way parallelism for a single query at the shard level, regardless of how many cores are available. Linear scaling: up to the number of physical cores, if the workload is CPU-bound and there is enough work to parallelize. Plateau: as cores saturate, throughput stops improving; latency may still improve slightly for a single query. Degradation: oversubscription causes context switching, cache thrashing, and memory-bandwidth contention, reducing throughput. Parallelism ceiling: bounded by the number of segments and shards on the node, not just by core count. I/O-bound case: on on-disk collections, more threads can increase queue depth and help until the disk saturates. Concurrency vs parallelism: more concurrent client requests are not the same as more threads per request; they compete for the same cores. The trade-off is throughput against latency and fairness. More threads can reduce the latency of a single query if there is work to parallelize, but they can also increase the latency of other queries by stealing cores. If the node is also running the optimizer or handling writes, the search threads compete with those workloads, and oversubscribing makes the contention worse. My rule is to size the search thread pool close to the physical core count, leave headroom for the optimizer and for the OS, and measure throughput and p99 under representative load rather than under a single-query benchmark. The common mistake is benchmarking a single query with high thread count and concluding that high thread count is good, without measuring the system under concurrent load. The second mistake is assuming that adding threads will reduce latency for a single query when the collection has few segments or shards - there is simply not enough parallel work. The third mistake is ignoring the optimizer: on a node where the optimizer is also running, the effective core count for search is lower than the total, and oversubscribing amplifies the contention. Version note: how Qdrant exposes and uses search thread configuration has changed across releases; in some versions it is a server-level setting, in others it is per-request, and the default behavior differs. # Thread configuration is a server-level setting in most Qdrant versions.
# The tuning approach is empirical: measure throughput and p99 at several
# thread counts under representative concurrent load.

from qdrant_client import QdrantClient
import concurrent.futures
import time

client = QdrantClient('http://localhost:6333')

def run_load(concurrency, queries):
    def one(q):
        t0 = time.perf_counter()
        client.query_points(collection_name='docs', query=q, limit=10)
        return (time.perf_counter() - t0) * 1000

with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as ex:
        lat = list(ex.map(one, queries))
    lat.sort()
    return {'p50': lat[len(lat)//2], 'p99': lat[int(len(lat)*0.99)-1]}

# Sweep concurrency at a fixed query set and watch where p99 turns up.
# The knee is usually near the physical core count minus optimizer headroom. Version-dependent: the configuration surface for search threads (server-level vs per-request, the default value, and whether the optimizer shares the same pool) has c

What is the effect of increasing the number of search threads/parallelism on a single node with limited CPU cores?

Parallelism helps until cores and memory bandwidth are saturated, then hurts

What is the effect of increasing the number of search threads/parallelism on a single node with limited CPU cores?

Parallelism helps until cores and memory bandwidth are saturated, then hurts

Scenario Questions

Follow-up Questions