Optimizing RAG Retrieval Accuracy Using Hybrid Search and Reranking Pipelines

Vector-only search is a trap that leads to "hallucinations by omission"—where the LLM fails not because it's stupid, but because the retrieval step missed the exact chunk of documentation needed. I spent the last month fighting with a RAG pipeline for a technical codebase where semantic search kept returning "similar" concepts instead of the "exact" API reference. Switching to a hybrid approach with a reranking layer solved about 80% of these misses.

The core issue is that embeddings are great for concepts but terrible for specific keywords or unique IDs. If you search for UserAuthService, a vector search might give you AuthManager because they are semantically close, but if you need the specific logic in UserAuthService, you're out of luck.

Here is the architecture that actually works: BM25 (Keyword) + Vector (Semantic) → Reciprocal Rank Fusion (RRF) → Cross-Encoder Reranker.

For the retrieval part, I use Qdrant because it handles both dense and sparse vectors in one collection. The trick is the RRF step, which merges the two result lists without needing to normalize the scores (since BM25 and Cosine Similarity are on different scales).

# Example logic for merging results via RRF
def reciprocal_rank_fusion(results_list, k=60):
    scores = collections.defaultdict(float)
    for results in results_list:
        for rank, doc_id in enumerate(results):
            scores[doc_id] += 1.0 / (rank + k)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

But the real magic happens at the reranking stage. RRF gives you a broad set of candidates, but it doesn't actually "understand" the relationship between the query and the document; it just looks at rank positions. I've integrated BGE-Reranker-v2-m3 as a final filter. A Cross-Encoder looks at the (Query, Document) pair simultaneously, which is computationally expensive but incredibly accurate.

I’ve automated the prompt refinement for this pipeline using Cursor's @Codebase feature. Instead of manually writing retrieval prompts, I use a "Query Expansion" step where the LLM generates three variations of the user's question to cast a wider net during the initial hybrid search.

My current config tips for production:

Top-K Strategy: Pull 50-100 documents during the hybrid phase, then prune down to the top 5 using the reranker. This catches the "long tail" of relevant data that vector search often misses.

Chunking Strategy: Stop using fixed-size chunks. Use "Semantic Chunking" or split by function/class boundaries in code. Rerankers struggle when a chunk is cut off mid-sentence.

Latency Trade-off: Reranking adds 100-300ms. If your app needs sub-second responses, only trigger the reranker if the top vector score is below a certain confidence threshold.

One major gotcha: be careful with your BM25 weights. If your dataset is small, BM25 can easily overpower the vector search, turning your RAG into a basic keyword search. I usually weight the vector score higher (0.7) and the keyword score lower (0.3) unless I'm dealing with highly specific technical jargon.

If you're still relying on a simple vector_store.similarity_search(), you're leaving a massive amount of accuracy on the table. The combination of Hybrid + Rerank is the only way to get RAG to a "production-ready" state for complex domains.