How would you decide the initial number of shards for a new collection when the eventual data size is uncertain?

The shard count is set at collection creation and cannot be changed without re-sharding, which is a heavier operation than adding replicas or moving shards. So the initial count must be planned generously for the eventual size, not just the current size. The rule of thumb is to size for 2-3 years of growth, or for the upper bound of the size estimate, and to over-provision rather than under-provision. If the eventual data size is uncertain, choose a shard count that is comfortable for the high end of the estimate: for a collection that might reach 500M points, 16 shards is a reasonable starting point (each shard holds ~30M points, which fits comfortably on a node). If the collection might reach 50M points, 4-8 shards is sufficient. The cost of over-provisioning is a small increase in fan-out overhead and coordination cost; the cost of under-provisioning is a re-sharding operation, which requires migrating data, may cause downtime, and is operationally complex. Over-provisioning is almost always the right choice when the eventual size is uncertain.

The mechanism behind this rule is that the shard count determines the parallelism of the collection and the granularity of data placement, but it also determines the fan-out cost of every query. More shards means more parallel work but more coordination and merge overhead. The sweet spot depends on the data size and the query pattern. For a small collection, a single shard is fine; for a large collection, many shards are necessary to distribute the data and the query load. The uncertainty is about the eventual size, so the decision should be based on the upper bound. There is a second consideration: custom sharding. If the collection uses custom sharding by tenant, the number of shards is the number of shard keys (or a multiple), and the initial count should accommodate the eventual tenant count. This is different from hash sharding, where the shard count is a global parameter. The third consideration is the replication factor, which multiplies the storage and compute requirements and should be planned separately from the shard count. Adding a replica is cheap compared to re-sharding, so the replication factor can be adjusted later.

The trade-off is between over-provisioning (small extra overhead) and under-provisioning (potential re-sharding). The common mistakes are: (1) choosing the shard count based on the current size, which leaves no room for growth; (2) assuming that shard count can be changed easily, which it cannot; (3) not considering custom sharding's different model; (4) choosing too many shards for a small collection, which adds overhead without benefit; (5) not planning for the replication factor at the same time. Version note: the re-sharding and shard-migration capabilities have evolved across Qdrant releases. Some versions support more flexible shard management; others require a full rebuild. Verify the version's capabilities before deciding, because the cost of under-provisioning depends on how hard it is to re-shard.

Version-dependent: the shard count API and the re-sharding capabilities have changed across Qdrant releases. Some versions expose tools for moving shards or changing the shard count; others require a full rebuild. Custom sharding is a recent addition. Verify the version's support for shard management before deciding the initial count.