How would you evaluate whether a change to your recommendation retrieval pipeline actually improved results, before rolling it out to all users?

The evaluation has two stages: offline and online. Offline, you compute relevance metrics on a held-out dataset using a ground-truth definition of relevance. The standard metrics are nDCG (normalized discounted cumulative gain) and MRR (mean reciprocal rank) for ranking quality, and recall@k for coverage. For recommendation, the ground truth is typically derived from user interactions - items the user engaged with in the future are treated as relevant, and the metrics measure how well the new pipeline ranks those items. The offline evaluation is fast, cheap, and lets you compare many variants, but it has a well-known limitation: it measures ranking quality against a fixed ground truth, which may not reflect the actual user experience or the business metrics that matter. A pipeline can improve nDCG while hurting click-through, so offline results are necessary but not sufficient. The online stage is a staged A/B test: the new pipeline is deployed to a small fraction of traffic, and the metrics that matter (click-through, conversion, retention, revenue) are compared with the control group. The staged approach limits the blast radius of a bad change and gives a statistically sound comparison.

The mechanism that makes this work is that the two stages answer different questions. Offline evaluation answers 'does the new pipeline rank relevant items higher on the historical data?' Online evaluation answers 'does the new pipeline produce better outcomes for real users?' The offline stage filters out clearly bad changes and lets you iterate quickly; the online stage validates the ones that pass. The offline ground truth is the key design decision: if it is derived from future interactions, it captures the user's actual preferences; if it is derived from a hand-labeled set, it captures expert judgment but may not reflect the user population. The online experiment must control for confounding - the treatment and control groups must be sampled from the same population, the time period must be the same, and the metrics must be pre-registered to avoid p-hacking. The experiment should run long enough to reach statistical significance, which depends on the effect size and the traffic volume. A common practice is to run for at least one week to capture weekly patterns, and to use a sequential test or a fixed-horizon test with a pre-computed sample size.