Optimizing RAG Performance Using Hybrid Search and Re-ranking with BGE-Reranker

Vector search alone often fails when you're looking for specific keywords or technical terms that don't have strong semantic overlaps in the embedding space. I've spent the last month fighting with "hallucinations" in my internal documentation bot, only to realize the retriever was grabbing chunks that felt similar but lacked the exact technical identifiers I needed. Switching to a hybrid search (BM25 + Vector) paired with a BGE-Reranker completely fixed the precision issue.

The core problem with standard RAG is the "top-k" limitation. If your embedding model ranks the perfect answer at position 12, but you only feed the top 5 to the LLM, you've already lost. Hybrid search widens the net, and the re-ranker acts as the high-precision filter to bubble the truly relevant content to the top.

Here is the architecture I implemented using Qdrant and the BGE-Reranker model.

The Hybrid Retrieval Logic

Instead of relying solely on cosine similarity, I combine the dense vector score with a sparse BM25 score. I use a simple Reciprocal Rank Fusion (RRF) to merge these lists. This ensures that if a document is highly relevant in either the keyword search or the semantic search, it makes it into the "candidate pool."

Integrating BGE-Reranker

Once I have the top 20-30 candidates from the hybrid search, I pass them through BAAI/bge-reranker-v2-m3. Unlike embeddings, which represent documents as points in space, the re-ranker looks at the query and the document together to calculate a relevance score.

Here is a snippet of how I handle the re-ranking step in Python:

from SentenceTransformers import CrossEncoder

# Load the BGE Reranker model
reranker = CrossEncoder('BAAI/bge-reranker-v2-m3')

def get_relevant_context(query, candidate_docs):
    # candidate_docs is the list of results from Hybrid Search (BM25 + Vector)
    # We create pairs of (query, doc_text)
    pairs = [[query, doc.text] for doc in candidate_docs]
    
    # Get relevance scores
    scores = reranker.predict(pairs)
    
    # Sort documents by score in descending order
    scored_docs = sorted(zip(scores, candidate_docs), key=lambda x: x[0], reverse=True)
    
    # Return only the top 5 most relevant chunks for the LLM context
    return [doc for score, doc in scored_docs[:5]]

Performance Tuning and Gotchas

The Latency Trade-off: Re-ranking adds overhead. Running a Cross-Encoder on 100 documents will kill your response time. Keep your initial hybrid retrieval window tight—usually between 20 and 50 documents. This keeps the re-ranking phase under 100ms on a decent GPU.

The "Lost in the Middle" Phenomenon: LLMs struggle when the answer is buried in the middle of a long context. By using BGE-Reranker to push the most critical information to the very top of the prompt, I noticed a significant drop in the LLM saying "I couldn't find the answer" when the answer was actually present in the retrieved chunks.

Config Tips for BGE: If you are deploying this in production, don't run the reranker on your main API thread. Wrap it in a lightweight FastAPI service or use a dedicated inference engine like Text Embeddings Inference (TEI) from Hugging Face to get the best throughput.

Concrete Productivity Gains

Precision: My "hit rate" at top-3 jumped from 62% to 88% on my internal test set.
Context Window: I can now send fewer, higher-quality tokens to Claude 3.5 Sonnet, which reduces costs and prevents the model from getting distracted by noise.
Robustness: The system now handles jargon and product IDs that the embedding model previously ignored.