Optimizing RAG Performance by Implementing Hybrid Search in Milvus Vector Database

Vector-only retrieval often fails when you're searching for specific product IDs, technical terms, or rare proper nouns because embeddings smooth out those distinct tokens into a general "semantic neighborhood." I hit this wall hard while building a technical documentation bot where users were searching for specific error codes like ERR_CONNECTION_RESET, and the vector search kept returning general "connection issue" articles instead of the exact match.

Switching to Hybrid Search in Milvus changed the game by combining Dense Vector search (for meaning) and Sparse Vector search (for keyword precision).

The trick to making this actually work is the Reranking step. If you just blindly merge results, the noise from the keyword search can pollute your context window. I've been using the Weighted Ranker, but for high-precision needs, a Cross-Encoder reranker is mandatory.

Here is how I set up the collection schema to support both:

from pymilvus import CollectionSchema, FieldSchema, DataType

# Define schema with both dense and sparse vectors
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True),
    FieldSchema(name="dense_vector", dtype=DataType.FLOAT_VECTOR, dim=768), # BERT/Ada
    FieldSchema(name="sparse_vector", dtype=DataType.SPARSE_FLOAT_VECTOR), # BM25/SPLADE
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535)
]
schema = CollectionSchema(fields)

To get the sparse vectors, I stopped relying on basic TF-IDF and moved to BM25 via the Milvus function feature. This allows the database to handle the sparse encoding internally, which cleans up my ingestion pipeline significantly.

When querying, I use the AnnSearchRequest to trigger both paths simultaneously. The crucial part is the RRFRanker (Reciprocal Rank Fusion), which balances the scores from both searches without requiring them to be on the same scale.

from pymilvus import AnnSearchRequest, RRFRanker

# Dense search request
dense_req = AnnSearchRequest(
    data=[query_dense_vector], 
    anns_field="dense_vector", 
    param={"metric_type": "L2", "params": {"nprobe": 10}}, 
    limit=10
)

# Sparse search request
sparse_req = AnnSearchRequest(
    data=[query_sparse_vector], 
    anns_field="sparse_vector", 
    param={"metric_type": "BIPARTITE", "params": {}}, 
    limit=10
)

# Hybrid search with RRF
results = collection.hybrid_search(
    reqs=[dense_req, sparse_req], 
    ranker=RRFRanker(), 
    limit=5
)

My current optimization stack for productivity:

• Cursor Rules: I added a .cursorrules file to my project specifying that any Milvus query logic must include a check for sparse_vector presence to prevent the AI from suggesting pure vector searches.
• Indexing: I found that HNSW for dense vectors is non-negotiable for latency, but keep an eye on memory usage as your collection grows; sparse vectors can bloat the index faster than you'd expect.
• The "Gotcha": Don't trust the default weights. If you're in a domain with heavy jargon (medical, legal, deep tech), lean harder on the sparse side.

The productivity gain is immediate: fewer "I don't know" responses from the LLM and significantly fewer hallucinations caused by the retriever fetching "semantically similar but factually wrong" chunks.