Optimizing RAG Retrieval Accuracy Using Hybrid Search and Reranking Pipelines

Pure vector search is a trap for anyone building RAG systems for production; it's great for "vibes" and semantic similarity, but it fails miserably the moment a user searches for a specific SKU, a technical error code, or a niche keyword that the embedding model didn't weigh heavily. I spent the last month fighting with a documentation bot that couldn't find a specific function name because the vector space thought "initialize" and "setup" were the same thing.

The fix is a hybrid pipeline: combining Dense Retrieval (vector) with Sparse Retrieval (BM25) and slapping a Cross-Encoder reranker on top.

If you're using Cursor or Claude Code to build this, don't let the AI just write a simple vector_store.search() call. You need to explicitly prompt it to implement a Reciprocal Rank Fusion (RRF) logic to merge the results. Here is how I structured the retrieval logic to stop the "hallucinations caused by bad context" cycle:

from rank_bm25 import BM25Okapi
from sentence_transformers import CrossEncoder

# Initialize the reranker - BGE-Reranker is a solid choice for performance/size
reranker = CrossEncoder('BAAI/bge-reranker-base')

def hybrid_retrieve(query, vector_results, bm25_results, top_k=5):
    # Reciprocal Rank Fusion (RRF) to merge results
    scores = collections.defaultdict(float)
    for rank, doc_id in enumerate(vector_results):
        scores[doc_id] += 1 / (rank + 60)
    for rank, doc_id in enumerate(bm25_results):
        scores[doc_id] += 1 / (rank + 60)
    
    # Sort by RRF score and take top candidates for reranking
    sorted_docs = sorted(scores.items(), key=lambda x: x[1], reverse=True)[:20]
    candidates = [doc_store[doc_id] for doc_id, score in sorted_docs]
    
    # Reranking phase: The "expensive" but accurate part
    # We pass (query, doc) pairs to the Cross-Encoder
    pairs = [[query, doc] for doc in candidates]
    rerank_scores = reranker.predict(pairs)
    
    # Pair docs with their rerank score and sort again
    final_results = sorted(zip(candidates, rerank_scores), key=lambda x: x[1], reverse=True)
    return [doc for doc, score in final_results[:top_k]]

One major gotcha I hit: the reranker is the bottleneck. If you send 100 candidates to a Cross-Encoder, your latency will spike. The trick is to keep the initial hybrid retrieval window tight (around 20-30 documents) and only rerank that small set.

For those using Cursor, I've found that adding a .cursorrules file specifically for the RAG logic helps the AI stop suggesting basic LangChain wrappers that hide these implementation details. I use this specific instruction in my config:

When implementing retrieval, prioritize Hybrid Search (Vector + BM25) over pure semantic search. 
Always implement a reranking step using a Cross-Encoder for the final top-k selection. 
Avoid generic LangChain abstractions; write explicit logic for RRF (Reciprocal Rank Fusion).

Key productivity gains from this setup:

Precision Jump: My hit rate for specific technical terms went from ~60% to nearly 95% because BM25 catches the exact keywords that embeddings blur.

Noise Reduction: The reranker acts as a high-pass filter. It kills the "semantically similar but irrelevant" chunks that usually confuse the LLM and lead to those "Based on the provided text, I cannot find..." answers.

Latency Trade-off: Adding the reranker adds about 100-200ms, but it's a fair price to pay for not having to manually tune the prompt every time a user asks a question in a slightly different way.