Optimizing Hybrid Search Performance using BGE-M3 and Milvus for RAG

BGE-M3 is a beast because it handles dense, sparse, and multi-vector representations in one go, but if you just dump it into Milvus with default settings, your latency will spike as your dataset grows. The real magic happens when you leverage Milvus's hybrid search capabilities to balance the precision of BM25-style keyword matching with the semantic depth of dense embeddings.

I've been benchmarking a RAG pipeline for a technical documentation site, and the biggest performance jump came from switching to a weighted reranking strategy rather than relying on a simple union of results.

To get this running, you need to set up a collection with two fields: one for the dense vector (BGE-M3's primary embedding) and one for the sparse vector (BGE-M3's lexical weights).

from pymilvus import CollectionSchema, FieldSchema, DataType

# Define schema for hybrid search
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True),
    # Dense vector for semantic search
    FieldSchema(name="dense_vector", dtype=DataType.FLOAT_VECTOR, dim=1024), 
    # Sparse vector for keyword/lexical search
    FieldSchema(name="sparse_vector", dtype=DataType.SPARSE_FLOAT_VECTOR), 
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535)
]
schema = CollectionSchema(fields)

The "gotcha" here is the index type. For the dense field, HNSW is the standard, but for the sparse field, you absolutely need to ensure you're using a compatible index to avoid full scans.

Performance Config Tips:

Dense Index: Use M between 16 and 32. If you go higher, your recall improves slightly, but your memory usage balloons.
Sparse Index: Stick to the default sparse index in Milvus 2.4+, but keep an eye on your ef parameter during search.
Reranking: Don't use a simple average. Use Reciprocal Rank Fusion (RRF) or a weighted score. I found that giving the dense vector a 0.7 weight and the sparse vector 0.3 weight works best for technical docs where specific terminology (like "CUDA" or "PyTorch") must be matched exactly.

Here is how I execute the hybrid search using the AnnSearchRequest to merge results:

from pymilvus import AnnSearchRequest, RRFRanker

# Dense search request
dense_req = AnnSearchRequest(
    data=[dense_embedding], 
    anns_field="dense_vector", 
    param={"metric_type": "L2", "params": {"ef": 64}}, 
    limit=100
)

# Sparse search request
sparse_req = AnnSearchRequest(
    data=[sparse_embedding], 
    anns_field="sparse_vector", 
    param={"metric_type": "IP"}, 
    limit=100
)

# Merge using RRF for balanced ranking
results = collection.hybrid_search(
    reqs=[dense_req, sparse_req], 
    rerank=RRFRanker(), 
    limit=10
)

One productivity gain I noticed: instead of manually managing the BGE-M3 model in a separate Flask wrapper, I integrated it using the FlagEmbedding library directly into my ingestion pipeline. It handles the generation of both dense and sparse vectors in a single call, which cuts my indexing time in half.

If you're seeing slow query times, check your ef value in the AnnSearchRequest. Increasing it improves recall but kills latency. Start at 64 and tune downwards until you hit the sweet spot for your specific hardware. For most of my workloads, ef=32 provides a 20% speedup with negligible loss in Top-10 accuracy.