Search vs Discovery: Why RAG often fails

Search retrieves artifacts; discovery reconstructs explanations. Most people conflate the two, but for anyone managing a complex codebase, the distinction is where the actual pain lies.

I recently hit a wall trying to figure out why a specific service was bypassing API gateway checks. I had the ADR, the Jira ticket, a Slack thread, and the config file all open. Search worked perfectly—it found every single one of those documents because they were all "relevant." But none of them actually answered the "why."

The answer didn't exist in a single document. It was scattered across a commit message, a legacy config line, and a random Slack conversation from six months ago.

This exposes a massive flaw in how we think about LLM agents and retrieval. Most AI workflows are built on the assumption that the answer lives inside one piece of text. You compute similarity, rank the top-k documents, and feed them to the LLM. This is great for factual lookups, but it's useless for organizational memory.

In a real-world AI workflow, the "unit of knowledge" isn't the document—it's the relationship between documents.

Search: "Where is the API doc?" (Solved by vector search)

Discovery: "Why does the system behave this way?" (Requires synthesizing fragments)

If you're building a custom RAG pipeline or a deep dive into your company's docs, stop measuring success by whether the system returned the "right document." Often, there is no single right document. The answer is the gap between the ADR and the actual implementation.

Until we shift from simple retrieval to actual discovery, we're just using expensive LLMs to summarize a bunch of incomplete fragments.