Optimizing Hybrid Search Accuracy with BGE Reranker and Milvus Vector Database

Pure vector search often falls short when you need exact keyword matching or specific domain terminology, which is why I've shifted my entire RAG pipeline to a hybrid approach using Milvus and BGE Reranker. The "vector-only" trap is real; you get results that are semantically similar but factually irrelevant because the cosine similarity doesn't care about specific identifiers or rare technical terms.

To get this working properly, I stopped relying on the basic hybrid search and implemented a two-stage retrieval process: coarse retrieval followed by a precision reranking step.

The Setup

I'm using Milvus 2.4+ because the native support for hybrid search (combining Dense and Sparse vectors) makes the plumbing much easier. The core logic is to fetch a wider net of candidates (say, top 100) using a weighted score of BM25 and Vector search, then pass those 100 through the BGE Reranker to prune them down to the top 5.

Implementation Logic

Here is the pattern I use to handle the reranking logic in Python. I use FlagEmbedding for the BGE model because it's the most stable wrapper for the cross-encoder.

from pymilvus import MilvusClient
from FlagEmbedding import FlagModel

# Initialize Milvus and BGE Reranker
client = MilvusClient("http://localhost:19530")
reranker = FlagModel('BAAI/bge-reranker-base', use_fp16=True)

def hybrid_search_with_rerank(query, collection_name, top_k=5):
    # 1. Hybrid Search: Retrieve a larger set of candidates
    # We use a weighted score to combine dense and sparse results
    results = client.hybrid_search(
        collection_name=collection_name,
        data=[query], 
        # Assuming a hybrid index is already configured
        # This retrieves the top 100 most likely candidates
        limit=100, 
        output_fields=["text"]
    )
    
    # Extract the text from the hits
    candidates = [hit['entity'].get('text') for hit in results[0]]
    
    # 2. Reranking: The 'Magic' step
    # BGE Reranker takes pairs of (query, document)
    pairs = [[query, doc] for doc in candidates]
    scores = reranker.predict(pairs)
    
    # Sort candidates by the reranker's score
    scored_docs = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    
    return [doc for doc, score in scored_docs[:top_k]]

Configuration Tips for Better Accuracy

Weighting the Hybrid Search: Don't just use a 50/50 split. In my experience with technical documentation, I set the sparse (keyword) weight higher (0.7) and the dense weight lower (0.3). This ensures that if a user searches for a specific error code like ERR_CONNECTION_RESET, that document surfaces regardless of semantic similarity.

The "Top-K" Gap: The gap between your hybrid retrieval limit (100) and your final rerank limit (5) is critical. If the correct answer isn't in the top 100 of the coarse search, the reranker can't save you. If you're seeing missing context, bump the coarse limit to 200, but be mindful of the latency hit on the reranker.

Hardware Gotchas: Running the BGE Reranker on a CPU is painfully slow for real-time apps. I moved the reranker to a separate small GPU instance (T4 is plenty) and wrapped it in a FastAPI endpoint. This decoupled the database latency from the model inference latency.

The Productivity Gain

Since implementing this, my "hallucination rate" dropped significantly because the LLM is no longer trying to make sense of "semantically similar but irrelevant" chunks. The BGE Reranker is incredibly good at spotting when a document actually answers the query versus just talking about the same topic. It’s the difference between getting a document that mentions "database optimization" and one that actually explains how to optimize a database.