Two teams store the same 50-million-vector collection - one keeps it fully in memory, one on disk with quantization. What operational differences should each expect?

The in-memory team should expect low and stable latency, simple capacity planning, and expensive hardware. Their p50 and p99 will be close together because every distance computation hits RAM. Their operational concerns are memory pressure - the collection must fit, including the graph and payload indexes, not just the raw vectors - and the cost of the machine. Their startup time is longer because the vectors have to be loaded and mapped, but steady-state latency is predictable. Their scaling is bounded by the largest machine they can buy, and vertical scaling is the only lever until they shard. The on-disk team should expect much cheaper hardware, a much smaller RAM footprint, and variable latency. Their p50 may be nearly as good as the in-memory team's if the working set is hot, but their p99 will be worse and more variable because cold pages require disk reads. Their operational concerns are page-cache management, disk performance (NVMe matters a lot), and the accuracy cost of quantization. They can fit the collection on a much smaller machine and can scale storage independently of RAM.

The mechanisms behind these differences are worth being explicit about. The in-memory team's latency is dominated by CPU and memory bandwidth, so tuning is about ef, m, and thread counts. The on-disk team's latency is dominated by page-cache hit rate and disk latency, so tuning is about quantization, inline storage, oversampling, and keeping the hot set resident. The in-memory team's failure modes are OOM and memory fragmentation; the on-disk team's failure modes are cache thrashing and latency cliffs when a background process evicts their hot pages. The in-memory team's cost is dominated by RAM; the on-disk team's cost is dominated by disk (especially NVMe) and by the CPU cost of rescoring, which is higher because the rescoring step runs full-precision distance computations. Both teams need to measure recall against ground truth, but the on-disk team's recall is affected by quantization and rescoring settings in addition to the usual HNSW parameters.