Optimizing RAG Retrieval Accuracy with LangChain Parent Document Retriever

Standard RAG often fails because there is a fundamental tension between retrieval and synthesis: small chunks are great for precise vector search (less noise), but they lack the surrounding context needed for the LLM to actually answer the question accurately. I spent a few days fighting with "hallucinations" in a technical documentation bot because the retrieved chunks were too fragmented to be useful.

The Parent Document Retriever in LangChain solves this by decoupling the document used for retrieval from the document used for synthesis. It stores large "parent" documents but indexes smaller "child" chunks. When a child chunk hits a similarity match, the system fetches the entire parent document for the LLM.

Here is the architectural setup I use to get this running:

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain_community.vectorstores import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings

# The 'parent' splitter keeps enough context for the LLM
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
# The 'child' splitter ensures high-precision vector search
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)

vectorstore = Chroma(collection_name="split_parents", embedding_function=OpenAIEmbeddings())
store = InMemoryStore() # In production, use Redis or a persistent DB

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

The biggest productivity gain here is the drastic reduction in "I don't have enough information" responses. By feeding the LLM a 2000-token parent block instead of a 400-token snippet, the model can see the logical flow of the documentation.

A few hard-won tips for configuring this:

The Child-to-Parent Ratio
Don't make the gap too wide. If your child chunks are 100 tokens and your parents are 10,000, you're basically just doing "document retrieval," which introduces too much noise and can blow out your token window. A 1:5 ratio (e.g., 400 child / 2000 parent) is usually the sweet spot for technical manuals.

Memory Leaks with InMemoryStore
If you are using InMemoryStore for testing, remember that it wipes every time the kernel restarts. For actual projects, swap this for RedisStore. If you don't, you'll spend half your morning re-indexing your PDF library.

Overlapping is still key
Even with the parent-child split, I still apply a small overlap (about 10-15%) to the child splitter. This prevents the vector store from "cutting a sentence in half" and missing a keyword that would have triggered the retrieval of the parent document.

Gotcha: Indexing Time
Be aware that indexing takes longer because LangChain is effectively managing two different sets of data. If you're dealing with millions of rows, the overhead of mapping child IDs to parent IDs in the docstore becomes noticeable.

Using Cursor's @Codebase feature to refactor my existing RAG pipeline into this structure took about 20 minutes because the logic is straightforward once you stop treating chunks as the final unit of delivery. It turns the retrieval process from a "keyword hunt" into a "context retrieval" system.