Optimizing Milvus index types for low-latency RAG retrieval at scale

DeepSeek-V3 and GPT-4o have pushed the boundaries of context windows, but relying on "long context" is a latency trap for production RAG. When your vector store hits the million-scale mark, the gap between HNSW and IVF_FLAT becomes a critical bottleneck for your p99 latency. I've spent the last month benchmarking these index types in Milvus to find the sweet spot where retrieval speed doesn't kill the user experience.

The trade-off is essentially a battle between memory overhead and search speed. HNSW (Hierarchical Navigable Small World) is the gold standard for low latency because it avoids the coarse quantization phase. In my tests with 5M vectors (768 dimensions), HNSW kept query times under 15ms, but the RAM usage was staggering. If you're running on a tight budget, HNSW will eat your cluster alive because it stores the graph structure in memory.

IVF_FLAT is the safer bet for memory efficiency, but it introduces a "cluster" problem. You have to tune nlist carefully. If nlist is too low, you're scanning too many vectors per probe; if it's too high, you lose recall. I found that increasing nprobe helps recover recall, but it linearly increases latency. For the same 5M dataset, IVF_FLAT with a high nprobe jumped to 40-60ms, which is noticeable when you're chaining multiple RAG steps.

For those pushing into the 10M+ range, IVF_PQ (Product Quantization) is where things get interesting. It compresses the vectors, meaning you can fit massive datasets into a smaller memory footprint. The catch is the precision loss. When I compared the retrieval results of IVF_PQ against HNSW using a standard RAG evaluation set, I saw a 3-5% drop in Top-10 recall. For most semantic searches, that's negligible, but for highly specific technical documentation, it can lead to the LLM hallucinating because the retrieved context was "close enough" but technically wrong.

Here is the general logic I used for the index configuration in my Python client:

# Example: Optimizing for high-recall, low-latency (HNSW)
index_params = {
    "metric_type": "L2", 
    "index_type": "HNSW", 
    "params": {"M": 16, "efConstruction": 64}
}

# Example: Optimizing for memory efficiency at scale (IVF_PQ)
index_params_pq = {
    "metric_type": "L2", 
    "index_type": "IVF_PQ", 
    "params": {"nlist": 2048, "nbits": 8}
}

Performance Breakdown:

HNSW: The Latency King. Best for datasets under 10M where RAM isn't the primary constraint. It offers the fastest query speeds and highest recall but is the most "expensive" in terms of hardware.

IVF_FLAT: The Balanced Middle. Good for medium-scale datasets. It's faster to build than HNSW and uses less memory, but query latency is highly dependent on the nprobe setting.

IVF_PQ: The Scale Specialist. Mandatory for datasets that exceed available RAM. It leverages compression to keep latency stable at massive scales, though you sacrifice a bit of accuracy.

If you're building a RAG pipeline where the LLM is already taking 2 seconds to generate a response, an extra 30ms in Milvus isn't the end of the world. However, if you're doing agentic workflows with 10+ retrieval loops per request, the cumulative latency of IVF_FLAT becomes a dealbreaker. Go with HNSW if you can afford the RAM; otherwise, tune IVF_PQ and accept the slight recall hit.