Building a Custom RAG Pipeline Using Doubao LLM and LangChain

Doubao's API has become a surprisingly viable alternative for RAG pipelines in domestic projects, especially when paired with LangChain's ecosystem. The biggest hurdle isn't the code—which is straightforward—but managing the chunking strategy to avoid the "lost in the middle" phenomenon that often plagues long-context retrieval.

To get this running, you first need to wrap the Doubao model via the OpenAI-compatible interface. Since Doubao uses a specific endpoint structure, the trick is to ensure your base_url is mapped correctly in the ChatOpenAI instantiation.

from langchain_openai import ChatOpenAI
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA

# Doubao API configuration
llm = ChatOpenAI(
    model="doubao-1-pro-32k", 
    openai_api_key="your_doubao_api_key",
    openai_api_base="https://ark.cn-beijing.volces.com/api/v3"
)

# Using a local embedding model to keep costs down and speed up retrieval
embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-small-zh-v1.5")

One thing I've noticed is that standard character-based splitting often breaks the semantic meaning of technical Chinese documentation. I've switched to a RecursiveCharacterTextSplitter with a specific overlap. If your overlap is too small, the LLM loses the context of the previous paragraph; too large, and you waste tokens on redundant data.

My current sweet spot for technical docs:

• chunk_size: 500
• chunk_overlap: 80
• separators: ["\n\n", "\n", "。", "！", "？", " ", ""]

Once the documents are indexed in FAISS, the real challenge is the prompt. Doubao is quite sensitive to how instructions are phrased. If you use a generic "Answer based on the context" prompt, it sometimes hallucinates or ignores the provided snippets if it thinks its internal knowledge is more "correct."

I found that forcing a "strict mode" in the prompt works best:

from langchain.prompts import PromptTemplate

template = """You are a precise technical assistant. Use ONLY the following pieces of retrieved context to answer the question. 
If the answer is not in the context, explicitly state that you do not know; do not try to make up an answer.

Context: {context}
Question: {question}

Answer:"""

QA_CHAIN_PROMPT = PromptTemplate.from_template(template)

qa_chain = RetrievalQA.from_chain_type(
    llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3}),
    chain_type_kwargs={"prompt": QA_CHAIN_PROMPT}
)

A major gotcha: Doubao's token counting differs slightly from GPT-4, so if you're building a production-grade pipeline, don't rely on simple character counts for your windowing logic. Always use a proper tokenizer to avoid 400 errors on long prompts.

Productivity-wise, using Cursor's @Codebase feature to iterate on this pipeline saved me hours. I just fed the LangChain documentation into the index, and it handled the boilerplate of the RetrievalQA chain implementation perfectly. The combination of a fast local embedding model (BGE) and Doubao's inference speed makes the end-to-end latency acceptable for a real-time chat interface.