RAG Performance: Why Ranking Isn't Your Real Problem
The "Different Tie" Syndrome
I was chasing a specific query: "Why does the gateway reject service X?"
- BM25: Top result was a glossary definition of "gateway." (Keyword match, zero intent).
- Hybrid Search: The glossary dropped, and a Jira ticket moved up. (Symptom match, no cause).
- Cross-Encoder: The Jira ticket hit #1. (Still no answer).
- New Embedding Model: The actual architecture doc—which contained the answer—finally crawled up to #5.
I felt like I was making progress because the documents were moving. In reality, I was just polishing a list of irrelevant candidates.
Selection vs. Ranking
The hard truth is that no reranker can promote a document that never made it into the candidate set. If your retrieval stage (selection) fails to pull the correct chunk, your ranking stage is just deciding which "wrong" answer looks the most confident.
I spent way too long asking, "How do I rank these better?" instead of asking, "Why are these the only candidates?" This is where most AI workflow bottlenecks actually happen.
The Retrieval Hierarchy
To avoid this, I've started treating retrieval as a three-layer stack. If you're doing a deep dive into your LLM agent's performance, check these in order:
1. Selection: The initial filter. What actually becomes a candidate? (This is where most failures happen).
2. Assembly: How is the data chunked, structured, and connected? (Poor chunking kills selection).
3. Ranking: The final sort. In what order do the candidates appear?
If your "Selection" layer is broken, the "Ranking" layer is useless. Most of us obsess over the third layer because it's where the fancy models live, but the first layer is what actually determines if the answer is even possible.
Stop polishing the order of documents that should never have been candidates in the first place.