Optimizing Vector Database Indexing for Faster Semantic Search in RAG Pipelines

HNSW (Hierarchical Navigable Small World) is the gold standard for most of us running RAG pipelines, but if you're just using the default settings in Milvus or Pinecone, you're likely leaving a lot of latency on the table. I spent the last few days tuning a search index for a technical documentation bot, and the difference between "default" and "optimized" was a jump from 120ms to 15ms per query.

The biggest trap is the M and efConstruction parameters. Most people ignore these, but M (the maximum number of connections per node) directly impacts the recall-latency trade-off. If your data is high-dimensional (like using text-embedding-3-large), a low M will cause the search to miss the actual nearest neighbors because the graph isn't connected enough.

For my current setup, I've found that bumping M to 32 and efConstruction to 200 during the indexing phase significantly stabilizes the recall. The trade-off is that indexing takes longer and consumes more RAM, but for a production RAG pipeline, that's a price worth paying.

When it comes to the actual query time, ef (the search-time exploration parameter) is your primary lever. I use Cursor to script a quick benchmarking loop to find the "elbow point" where increasing ef no longer improves recall but continues to spike latency.

Here is the Python snippet I use to stress-test the recall vs. latency balance:

import time
from pymilvus import Collection

collection = Collection("doc_embeddings")

def benchmark_ef(ef_values):
    results = []
    for ef in ef_values:
        start = time.perf_counter()
        # Search with specific ef value
        res = collection.search(
            data=[query_vector], 
            anns_field="vector", 
            param={"ef": ef}, 
            limit=5
        )
        latency = (time.perf_counter() - start) * 1000
        results.append((ef, latency))
    return results

# Test range to find the sweet spot
print(benchmark_ef([10, 20, 40, 64, 128]))

One "gotcha" that bit me recently: Product Quantization (PQ). I tried enabling PQ to save memory, but the precision drop was noticeable in the RAG output—the LLM started hallucinating because the retrieved chunks were "close enough" in the compressed space but irrelevant in reality. If you have the RAM, stick to FLAT or HNSW without aggressive quantization. If you must use PQ, try IVF_SQ8 as a middle ground.

To make the pipeline actually feel snappy, I also implemented a two-stage retrieval process. I use a fast, coarse-grained index to pull 50 candidates, then use a cross-encoder (like BGE-Reranker) to narrow it down to the top 5. This is way more effective than trying to force a vector index to be 100% precise.

My current production config for high-precision RAG:

• Index Type: HNSW
• M: 32 (Balanced connectivity)
• efConstruction: 200 (Better graph quality)
• ef (Search): 64 (Low latency, high enough recall)
• Reranker: BGE-Reranker-v2-m3 (To fix the "top-k" noise)

If you're seeing "drift" in your search results over time as your dataset grows, check your index fragmentation. Rebuilding the index from scratch once a month is often faster than dealing with the performance degradation of incremental updates in some vector DBs.