What would you include in a regression test suite to catch an accidental change in distance metric or embedding model during a refactor?

The regression test suite should encode the expected relationships between a fixed set of embeddings, so that any change to the distance metric or the embedding model produces a failure. The core of the suite is a small set of hand-picked texts whose semantic relationships are known and stable: a query and its exact match, a query and a paraphrase, a query and an unrelated text, and a query and an antonym or a negation. For each pair, the test asserts a specific ordering or a threshold on the similarity. For example: the exact match must rank first, the paraphrase must rank above the unrelated text, and the unrelated text must fall below a similarity threshold. These assertions are independent of the specific embedding model, so they remain valid as long as the model is semantically correct. If the model changes to one that does not preserve the relationships, or if the distance metric changes in a way that alters the ranking, the test fails.

The mechanism that makes this catch regressions is that the test encodes the semantic invariants of the system rather than the exact numerical output. An exact-output test would break on every model upgrade and would be too brittle to maintain. An invariant-based test survives model upgrades that preserve semantics but fails on changes that break them. To catch a distance-metric change specifically, include a test that computes similarity under the expected metric and compares with Qdrant's output. To catch an embedding-model change, include a test that verifies the dimension of the stored vectors and a test that verifies a known similarity ranking. If the collection's metric is changed from cosine to dot product, the ranking of normalized vectors should be similar but not identical; the test should be tight enough to detect the difference if the vectors are not normalized. If the embedding model is changed, the vector dimension will almost certainly change (catching the error immediately) but even if it does not, the ranking of the fixture texts will change and the test will fail.

The trade-off is between tightness and maintainability. A tight test that asserts exact similarity values catches every change but breaks on every model upgrade, which is unmaintainable. A loose test that asserts only 'the exact match is in the top 5' is maintainable but misses subtle regressions. The right balance is to assert on orderings and thresholds that are robust to model changes but sensitive to semantic errors. The common mistake is to assert on exact numerical similarity, which makes the test brittle and causes developers to disable it. The second mistake is to use the production collection or production data, which makes the test slow and non-deterministic. The third mistake is to test only the happy path (the exact match is found) and not the failures (the unrelated text is not ranked above the match). The fourth mistake is to forget the dimension assertion, which is the cheapest and most direct way to catch a model change. Version note: the distance metric options and their semantics have been stable, but the query API has changed - query_points replaced search in qdrant-client 1.10+ - so the test code may need updating when the client is upgraded, even if the underlying behavior is unchanged.

Version-dependent: the query API and the collection info response have changed across client versions. The query_points method is qdrant-client 1.10+; older clients used search(). The exact shape of the config.params.vectors in the collection info response may differ. The distance metric semantics (cosine, dot, Euclidean) have been stable, but the availability of some metrics in combination with quantization or multivector fields has evolved.