What problem does vector quantization solve, and what is the fundamental trade-off it introduces?

Vector quantization solves the memory problem. A single float32 vector of 768 dimensions is 3072 bytes. A million of them is about 3 GB just for the raw vectors, before any index overhead. At 100 million vectors you are at 300 GB, which is more than most machines have and far more than fits comfortably in RAM. Quantization compresses each vector to a lower-precision representation - int8, or a few bits per dimension, or even one bit per dimension - which reduces the memory footprint by 4x to 32x depending on the scheme. The second, less obvious benefit is speed: lower-precision vectors fit more values per SIMD register, so distance computations are faster in addition to touching less memory. On modern CPUs, int8 dot products can be several times faster than float32 for the same number of dimensions, and binary dot products are faster still because they reduce to popcount operations.

The trade-off is accuracy. Quantization is lossy: you are approximating each vector with a lower-precision surrogate, and the distance you compute is an approximation of the true distance. The error is small for scalar quantization (int8) because 256 levels per dimension is enough to preserve the geometry of most embeddings, but it grows as you compress more aggressively. With binary quantization, each dimension is reduced to a single bit (sign), which is a very coarse approximation; the ranking it produces is noisy, and you need to retrieve more candidates and rescore them with the original vectors to recover accuracy. The fundamental trade-off is therefore memory and speed against recall. There is no quantization scheme that is free; you are always giving up some accuracy to gain some efficiency, and the right choice depends on how much accuracy your application can tolerate and how much memory you have. The mistake less experienced engineers make is assuming quantization always reduces recall. It can, but on many real datasets the recall drop with int8 scalar quantization is under one point, while memory drops 4x - a very favorable trade. The opposite mistake is assuming quantization is always safe and skipping the recall measurement entirely, which is how teams ship a 10-point recall regression without noticing.