Optimizing RAG Retrieval Performance Using Hybrid Search in Milvus Vector Database

Vector-only retrieval often fails on specific keywords or product IDs, which is why I've pivoted my RAG pipeline to Hybrid Search in Milvus. Relying solely on cosine similarity with embeddings leads to "semantic drift"—where the AI finds things that are conceptually similar but factually wrong because it missed a specific technical term.

The real power comes from combining Dense vectors (for meaning) and Sparse vectors (for keyword matching) using a Reciprocal Rank Fusion (RRF) algorithm. This ensures that if a document ranks high in either search method, it bubbles up to the top.

To get this running, you need to define a schema that supports both vector types. I found that using BM25 for the sparse part is the gold standard here. Here is how I set up the collection:

from pymilvus import MilvusClient, DataType

client = MilvusClient(uri="http://localhost:19530")

# Define schema for Hybrid Search
schema = client.create_schema(auto_id=True)
schema.add_field(field_name="id", datatype=DataType.INT64, primary_key=True)
schema.add_field(field_name="dense_vector", datatype=DataType.FLOAT_VECTOR, dim=768) # BGE-small
schema.add_field(field_name="sparse_vector", datatype=DataType.SPARSE_FLOAT_VECTOR) 
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=65535)

# Indexing is critical: HNSW for dense, SPARSE_INVERTED_INDEX for sparse
index_params = {
    "index_type": "HNSW", 
    "metric_type": "L2", 
    "params": {"M": 16, "efConstruction": 64}
}
client.create_index(collection_name="rag_docs", field_name="dense_vector", index_params=index_params)
client.create_index(collection_name="rag_docs", field_name="sparse_vector", index_params={"index_type": "SPARSE_INVERTED_INDEX", "metric_type": "IP"})

When querying, don't just run two separate searches and manually merge them. Use the AnnSearchRequest and WeightedRRF to let Milvus handle the fusion at the engine level. This reduces latency significantly.

from pymilvus import AnnSearchRequest, WeightedRRF

# Create search requests for both vectors
dense_req = AnnSearchRequest(
    data=[query_dense_vector], 
    anns_field="dense_vector", 
    param={"radius": 10}, 
    limit=10
)
sparse_req = AnnSearchRequest(
    data=[query_sparse_vector], 
    anns_field="sparse_vector", 
    param={"radius": 10}, 
    limit=10
)

# Use RRF to fuse the results
res = client.hybrid_search(
    collection_name="rag_docs",
    reqs=[dense_req, sparse_req],
    rerank=WeightedRRF(weight_dense=0.7, weight_sparse=0.3),
    limit=5
)

A few hard-learned tips for those implementing this:

Weight Tuning
I started with a 50/50 weight, but my recall was messy. For technical documentation, I found a 0.7 (dense) / 0.3 (sparse) split works best. The dense vector handles the intent, while the sparse vector acts as a "safety net" for exact terminology.

The Tokenization Trap
If you're using a separate library for sparse vector generation (like pymilvus[model]), ensure the tokenizer used for indexing is identical to the one used for querying. If you use a different stemmer or tokenizer for the query, your sparse match rate will plummet to zero without any error messages.

Memory Overhead
Adding sparse vectors increases the memory footprint. Monitor your RAM closely. If you're hitting limits, try reducing the efConstruction in your HNSW index for the dense vectors to trade a bit of accuracy for a lot of memory.

The productivity gain here is massive. I stopped spending hours tweaking my embedding model's fine-tuning and instead just fixed the retrieval logic. It turned my "almost correct" RAG bot into one that actually finds the exact line of code or documentation page required.