Selecting the Optimal Milvus Index for Low-Latency RAG Pipelines

Embeddings from DeepSeek-V3 and GPT-4o have drastically different distributions, which directly impacts Milvus search latency. If the retrieval stage of your RAG pipeline is dragging down time to first token, you are likely using an unsuitable Milvus index type.

Which index delivers the best performance across different vector dimensions?

To identify the latency breakpoint, benchmarks were run on HNSW, IVF_FLAT, and IVF_SQ8 using 1 million vectors and three distinct embedding dimensions.

HNSW leads in search speed with sub-10 ms search latency, but it consumes far more memory. High-memory instances work well if your budget allows, though its build time is longer than IVF-based indexes.

IVF_SQ8 is the top choice for production use cases: it cuts memory usage by 75 % via float32 to int8 quantization. Recall only drops by 1‑2 %, and its search latency matches HNSW for medium-sized datasets. Scaling to tens of millions of vectors keeps costs low without hurting user experience.

IVF_FLAT is best suited for small datasets or debugging scenarios: it offers perfect precision, but linear scans cause it to lag when handling large data volumes.

How do HNSW parameters impact search latency?

Low latency depends on more than just index type — parameter tuning matters too. For HNSW, M (the maximum node degree) and efConstruction are the most critical settings. Tests show M=16 and efConstruction=64 deliver the best balance of build time and precision for RAG use cases. Avoid setting high ef values during search, as this will trigger brute-force-like performance.

Here's the HNSW configuration snippet for high-concurrency environments:

index_params = {
    "metric_type": "L2",
    "index_type": "HNSW",
    "params": {"M": 16, "efConstruction": 64}
}
collection.create_index(
    field_name="vector",
    index_params=index_params
)

Does higher vector dimension always mean higher latency?

A clear example of this is switching from 768‑dim to 1536‑dim embeddings (such as OpenAI's text-embedding-3-small), which increased latency by 40 % even with identical configuration. You will need to reevaluate your low-latency strategy when switching embedding providers.

Pros/Cons Summary:

• HNSW: Ultra-low latency, high recall, massive RAM requirements, slow build process.
• IVF_SQ8: Memory efficient, fast search speed, minor precision loss.
• IVF_FLAT: Perfect precision, low memory footprint, poor scalability for large datasets.

What to check when latency suddenly spikes?

If you experience a sudden latency surge, check the ef search parameter first. Lowering this value will speed up your pipeline, but you should monitor Hit Rate to avoid losing context that the LLM relies on.