Exponential backoff with jitter and idempotent upserts
The right retry strategy is exponential backoff with jitter, combined with idempotent upserts and a bounded number of attempts. On a transient error (network timeout, 5xx, connection reset), the client waits a short interval, then retries. On each subsequent failure, the interval doubles, up to a maximum. Jitter adds a random component to the interval so that many clients that failed at the same time do not all retry at the same moment, which would create a thundering herd. The retry should be capped at a small number of attempts (e.g. 3-5) to avoid retrying forever on a persistent error. Upserts should be idempotent, which they are in Qdrant if you use deterministic point IDs: an upsert with the same ID overwrites the existing point, so retrying an upsert is safe. This is why using deterministic IDs (e.g. a hash of the content) rather than auto-generated IDs is a best practice - it makes retries safe and makes the write path robust to duplicate deliveries.
The mechanism that makes naive retry dangerous is that retries add load to a system that is already failing. If Qdrant is overloaded, every client retrying immediately doubles or triples the request rate, which makes the overload worse and can turn a transient degradation into a sustained outage. This is the classic retry storm. Exponential backoff reduces the load by spacing out retries, and jitter prevents synchronization. Bounded attempts prevent a client from retrying forever, which would tie up resources and mask the underlying problem. Idempotent upserts ensure that a retry that arrives after the first attempt actually succeeded does not create a duplicate point or corrupt data. Without idempotency, a retry after a timeout could create a duplicate or, worse, overwrite a newer version with an older one. The combination of backoff, jitter, bounded attempts, and idempotency is what makes the write path robust to transient failures without amplifying them.
Exponential backoff: double the wait on each retry, up to a maximum interval.
Jitter: add a random component to avoid synchronized retries.
Bounded attempts: cap at 3-5 attempts to avoid infinite retry loops.
Idempotent upserts: use deterministic point IDs so a retry overwrites rather than duplicates.
Distinguish error types: retry on transient errors (timeouts, 5xx); do not retry on client errors (4xx like dimension mismatch).
Circuit breaker: if a service is failing consistently, stop sending for a period to let it recover.
Observability: log retries and failures so that you can detect when retry rates spike.
The trade-off is between resilience and latency. A retry adds latency to the request, and with backoff the added latency grows with each attempt. For a latency-sensitive request, a retry may not be worth it; for a write that must not be lost, it is. The right policy depends on the operation: reads can often fail fast and let the caller decide, while writes may need to be durable. The common mistake is to retry immediately in a tight loop, which amplifies load and turns a transient issue into an outage. The second mistake is to retry non-transient errors (e.g. a dimension mismatch), which will never succeed and wastes resources. The third mistake is to use non-idempotent writes (auto-generated IDs) so that retries create duplicates. The fourth mistake is to not bound the retries, so a client can hang indefinitely. The fifth mistake is to not implement a circuit breaker, so a client keeps hammering a service that is clearly down. Version note: the qdrant-client has some built-in retry behavior in some versions, but the details and defaults have changed. If you rely on retries for correctness, implement them at the application level where you can control the policy.
Version-dependent: the qdrant-client's built-in retry behavior and the exception types it raises have changed across versions. Some versions have a retry configuration parameter; others do not. If your application depends on retries for durability, implement them explicitly and test them against a fault-injected environment (e.g. a proxy that drops connections) to confirm the behavior.
0-2 years experience
2-5 years experience
5-8 years experience
8+ years experience
