Optimizing Milvus index types for low-latency RAG retrieval at scale

HNSW is the gold standard for low-latency retrieval in Milvus, but if you're scaling to tens of millions of vectors, the memory overhead becomes a nightmare. I've spent the last month benchmarking HNSW against IVF_FLAT and IVF_PQ on a dataset of 50M vectors (768 dimensions) to see where the actual breaking point is for a production RAG pipeline.

The trade-off is always latency versus recall, but the gap between "theoretically fast" and "actually fast" depends entirely on your M and efConstruction settings. For those chasing sub-10ms retrieval, HNSW is unbeatable, but you're paying for it in RAM. My tests showed that while HNSW maintained a 98% recall rate, it consumed nearly 4x the memory of IVF_PQ.

HNSW (Hierarchical Navigable Small World)
Pros: Blazing fast query speed; extremely high recall; no need for a separate training phase.
Cons: Massive RAM consumption; slow index build time.
Performance: On my setup, query latency stayed under 5ms for 1k QPS, but the index size was bloated. If your budget allows for high-memory nodes, this is the only choice for RAG where precision is non-negotiable.

IVF_FLAT (Inverted File Flat)
Pros: Better memory efficiency than HNSW; faster build times.
Cons: Higher latency as nprobe increases; recall drops off sharply if nprobe is too low.
Performance: It's a middle-ground. To get close to HNSW's recall, I had to crank up nprobe, which pushed latency up to 25-40ms. It’s acceptable for internal tools, but feels sluggish for a user-facing chatbot.

IVF_PQ (Inverted File Product Quantization)
Pros: Tiny memory footprint; scales to billions of vectors.
Cons: Lossy compression means recall takes a hit; requires a training set to build the quantizer.
Performance: This is where you go when you're broke or your dataset is gargantuan. Latency is low, but the "precision drift" is real. In a RAG context, this occasionally retrieved irrelevant chunks that hallucinated the final answer because the vector was "close enough" in compressed space but wrong in reality.

If you're trying to tune your index, don't just stick to the defaults. For HNSW, I found that M=16 and efConstruction=64 provided the best balance of speed and accuracy for 768-dim embeddings. If you're using the Milvus Python SDK, your index params should look something like this:

index_params = {
    "metric_type": "L2",
    "index_type": "HNSW",
    "params": {"M": 16, "efConstruction": 64}
}
collection.create_index("embeddings", index_params)

For those running massive scales, I’d suggest a hybrid approach. Use IVF_PQ to narrow down the candidate set and then a re-ranking step with a Cross-Encoder. This offsets the recall loss of PQ while keeping the latency low.

The biggest mistake I see is people treating the index type as a "set and forget" configuration. As your collection grows, an HNSW index that worked at 1M vectors will either crash your pod or slow down your pipeline at 100M. Monitor your load_memory closely. If you see the memory usage spiking, it's time to evaluate if you can sacrifice 2% recall for the memory savings of IVF_PQ.