Optimizing RAG Retrieval Accuracy Using Hybrid Search and BGE Embeddings

Vector-only retrieval is a trap that makes RAG systems look great in demos but fail in production because semantic search often misses exact keyword matches—like a specific error code or a niche product ID. I've spent the last month swapping out standard OpenAI embeddings for BGE (BGE-M3 specifically) and implementing a hybrid search layer to fix this "semantic drift."

The biggest win came from moving to BGE-M3 because it's natively designed for multi-linguality and supports dense, sparse, and multi-vector retrieval in one model. If you're using a vector DB like Milvus or Qdrant, you can leverage this to run a hybrid query where you combine the cosine similarity of the dense vector with the BM25 score of the sparse vector.

Here is the logic I used to implement the Reciprocal Rank Fusion (RRF) to merge these results. RRF is essential because you can't simply add a dense score (0 to 1) to a BM25 score (which can be any positive number).

def reciprocal_rank_fusion(dense_results, sparse_results, k=60):
    scores = collections.defaultdict(float)
    
    # Rank dense results
    for rank, doc_id in enumerate(dense_results):
        scores[doc_id] += 1 / (rank + k)
        
    # Rank sparse results
    for rank, doc_id in enumerate(sparse_results):
        scores[doc_id] += 1 / (rank + k)
        
    # Sort by combined score descending
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

To get this working with Cursor, I stopped asking it to "write a retrieval function" and started providing it with the specific documentation for my vector DB's hybrid search API. The "gotcha" here is that if you let the AI guess the syntax for hybrid search, it often hallucinates an operator="OR" when the API actually requires a specific rerank parameter. I now keep a .cursorrules file that explicitly tells the AI: "When implementing search, always use RRF for hybrid merging and never assume default distance metrics; specify L2 or Cosine explicitly."

One configuration tip that actually moved the needle on accuracy: don't just embed the whole chunk. I started using a "Small-to-Big" retrieval strategy. I embed small sentences (the "child" chunks) using BGE-M3 for high-precision matching, but when a match is found, I feed the surrounding larger paragraph (the "parent" chunk) into the LLM context. This prevents the LLM from hallucinating due to lack of context while keeping the retrieval pinpoint accurate.

My current high-performance stack:

• Embeddings: BGE-M3 (hosted via Ollama or HuggingFaceTEI for low latency)
• Retrieval: Hybrid (Dense + BM25)
• Reranking: BGE-Reranker-v2-m3 (This is non-negotiable; if you don't rerank the top 20 results, you're leaving 15-20% accuracy on the table)
• Orchestration: LangGraph for the routing logic

The productivity gain is massive. Instead of spending hours tweaking the "top_k" parameter—which is a losing game—I now focus on the quality of the sparse indices. If the BGE dense vector misses a technical term, the BM25 side catches it, and the reranker puts it at the top. This setup virtually eliminated the "I can't find the document even though it's in the database" complaints from my team.