How would you validate that a newly restored cluster from snapshots is actually healthy and serving correct results before routing production traffic to it?

The validation has four layers: structural, index, functional, and end-to-end. Structural validation checks that the collection exists with the right configuration and that the point count matches the expected count from before the failure. Index validation checks that the HNSW graphs and payload indexes are built and that no segments are stuck in an unindexed state. Functional validation runs a set of known queries and asserts that the results match the expected IDs from a reference set - this is the strongest check because it verifies that the data is not just present but searchable and correctly indexed. End-to-end validation runs the actual application against the restored cluster in a shadow or canary mode, comparing the results with the expected behavior before cutting over. The validation should also check the payload schemas, the aliases, the replication factor, the shard placement, and the consistency settings, because a restore can miss metadata that is not part of the snapshot.

The mechanism that makes each layer effective is that it catches a different class of problem. A structural check catches a collection that was not restored or was restored with the wrong config. An index check catches a collection whose data is present but not indexed, which would cause queries to fall back to a brute-force scan and be slow. A functional check catches a collection whose index is built but whose data is wrong. An end-to-end check catches integration issues - the application connecting to the wrong collection, the wrong alias, a version mismatch. The layers are complementary: skipping any one leaves a class of failure undetected. The functional check is the most important because it is the closest to the user experience, but it requires a reference set of queries and expected results. Building that reference set before the failure is part of the preparation: you cannot validate after the fact without a known-good baseline.