Selecting the Best Milvus Index for Million-Scale Low-Latency RAG

DeepSeek-V3 and GPT-4o push context windows further, but in production retrieval augmented generation pipelines, long-context reliance often turns into a latency bottleneck. When a vector store reaches the million-scale mark, choosing between HNSW and IVF_FLAT directly shapes p99 latency.

The decision hinges on a memory versus speed trade-off. HNSW (Hierarchical Navigable Small World) sets the standard for fast retrieval by avoiding coarse quantization entirely. On a 5M vector dataset with 768 dimensions, HNSW kept query latency below 15ms, yet its RAM demand stays high because the full graph structure lives in memory. IVF_FLAT trades memory for scanning overhead by partitioning vectors into clusters, creating a tuning challenge around nlist. A low nlist forces more vectors per probe, while a high nlist lowers recall. Raising nprobe compensates recall but drives latency up linearly. Against the same 5M dataset, IVF_FLAT with elevated nprobe landed in the 40-60ms range.

IVF_PQ (Product Quantization) becomes necessary once datasets surpass 10M by compressing vectors to shrink memory footprint. This compression costs precision, and evaluating IVF_PQ against HNSW on a standard RAG benchmark revealed a 3-5% Top-10 recall drop. Though usually acceptable, that gap can amplify hallucinations in tightly scoped technical documentation. The index configuration logic in Python clients reflects these trade-offs directly.

Performance characteristics cluster around three profiles. HNSW dominates latency for datasets under 10M when RAM is not constrained, offering the quickest queries and strongest recall at higher hardware cost. IVF_FLAT balances memory savings against build speed, making it well suited for medium-scale datasets where query latency tolerates nprobe tuning. IVF_PQ targets scale experts handling datasets beyond available RAM, using compression to stabilize latency at the expense of measurable accuracy.

In agentic workflows running ten or more retrieval loops per request, IVF_FLAT latency compounds quickly enough to become a hard limit. HNSW remains the choice when RAM permits, while IVF_PQ offers a controlled fallback that accepts a small recall penalty to maintain throughput. https://milvus.io/docs/v2.4.x/index.md