Design a semantic search system that must support 500 million documents with sub-100ms p99 latency. What are the key architectural decisions?

The first decision is storage layout, because 500M documents at 768 dimensions of float32 is roughly 1.5 TB of raw vectors before any index. That will not fit in a single node's RAM at reasonable cost, and even if it did, the HNSW graph would add another 80-160 GB on top. The right answer is a hybrid: keep quantized vectors resident in RAM for traversal, keep the full-precision vectors on disk for rescoring, and put the HNSW graph on disk with inline quantized vectors to halve the I/O per traversal step. This combination makes sub-100ms p99 achievable without a huge RAM bill. The second decision is sharding: split the collection across enough shards that each shard fits comfortably on a node and each node can serve its share of the QPS. For 500M documents, 8-16 shards per replica set is a reasonable starting point, but the exact count depends on the query pattern and whether queries can be routed to a single shard via a shard key. The third decision is replication: for a mission-critical system, replication_factor=2 or 3 to survive a node failure without an outage. The fourth decision is caching: embedding cache at the application layer, result cache for repeated queries, and the OS page cache for the hot portion of the on-disk data. The fifth decision is the query path: use a two-stage pipeline with a cheap first-stage retriever and a reranker over a small candidate set.

The mechanism that lets this hit sub-100ms p99 is that each stage has a bounded cost. Quantized ANN traversal touches only the small quantized vectors, so it is memory-bandwidth-bound rather than disk-bound, and the graph is traversed in microseconds per hop. Rescoring touches only the top ~100 candidates with full-precision vectors, so the disk I/O is bounded and mostly cached. Sharding parallelizes the traversal across nodes, so the per-shard latency is a fraction of the total work, and the coordinator merges the per-shard top-k. Replication lets reads be distributed across replicas, which increases effective throughput and reduces the per-replica load. The p99 is dominated by the slowest shard and by the tail of the page-cache miss distribution, so the design must ensure that the working set fits in cache and that no single shard is a hotspot. The common failure mode is to underestimate the graph overhead and the payload index footprint, which pushes the working set out of cache and causes p99 spikes. Another common failure is to fan out queries to every shard on every request, which makes the coordinator and the slowest shard the bottleneck.