How would you approach re-embedding a multi-billion-point production collection with a new embedding model with zero search downtime?

Re-embedding a multi-billion-point collection is a multi-week project, and zero downtime requires a shadow collection that is populated in parallel with the live one. The pattern is: create the shadow collection with the new model's dimension and metric; dual-write new and updated points to both collections; backfill the existing points into the shadow collection by re-embedding them in batches; monitor progress and validate recall; cut over the query path to the shadow collection via an alias; keep the old collection for rollback. The backfill is the long pole: re-embedding a billion points takes a significant amount of GPU time and Qdrant write throughput, and it must be paced so it does not saturate the resources that the live collection needs. The dual-write ensures that points written during the backfill are present in both collections, and the backfill ensures that the historical points are present in the shadow collection. The cutover is a metadata operation (alias move) and can be rolled back quickly if the new model does not perform as expected.

The mechanism that makes zero downtime possible is that the live and shadow collections are independent, and the query path uses exactly one of them at a time. During the migration, the live collection serves all queries and the shadow collection is being populated. New writes go to both, so the shadow collection does not fall behind. The backfill processes the historical data in batches, which can be paused and resumed. The migration state is always consistent from the query path's perspective: it uses the live collection until the cutover, then the shadow collection. The cutover is atomic at the alias level, and the rollback is the reverse alias move. The main risks are: (1) the dual-write failing silently, so some points are missing from the shadow collection; (2) the backfill not completing before the cutover, so some points are missing; (3) the new model performing worse than the old one, which is why validation before cutover is essential; (4) the cutover causing a performance regression, which is why the shadow collection should be sized and configured to meet the SLO. The validation should include recall against a held-out set, ranking comparison with the old collection, and a load test of the shadow collection at production QPS.