Claude 3.5 Sonnet outperforms GPT-4o in long multi-step RAG pipelines by resisting context drift

Multi-step RAG pipelines often fail not because of vector database delays, but because chaining RunnablePassthrough and RunnableParallel introduces "reasoning drift"—where model responses degrade when retrieval outputs are fed sequentially into synthesis steps. Testing with a three-stage workflow (query expansion → multi-index retrieval → final synthesis) revealed that GPT-4o frequently hallucinated connections, distracted by expanded query terms appearing in retrieved documents. In contrast, Claude 3.5 Sonnet maintained stronger alignment with the original user intent across the same chain.

When generic RunnableSequence proves insufficient for complex logic, replacing it with a custom RunnableLambda provides better state visibility and logging control. For pipelines suffering from prompt leakage or lost context, explicitly structuring the chain with a dictionary in RunnableParallel ensures deterministic behavior by separating the original question from retrieved documents. The following pattern preserves the user’s input throughout execution:

from langchain_core.runnables import RunnablePassthrough, RunnableParallel

rag_chain = (
    RunnableParallel({
        "context": retriever | format_docs,
        "question": RunnablePassthrough()
    })
    | prompt
    | llm
    | StrOutputParser()
)

Model performance varies by chain depth:

• Claude 3.5 Sonnet excels in long, iterative pipelines where retrieval context risks overwhelming the model, though its slower Time to First Token (TTFT) may impact latency.
• GPT-4o delivers faster execution and handles concise, precise retrieval outputs well, but tends to prioritize recent context over initial instructions when retrieval results are verbose.
• DeepSeek-V2 performs cost-effectively in query expansion but struggles with strict LCEL formatting requirements, occasionally ignoring specified output schemas in later steps.

If the pipeline exceeds four steps, shifting from LangChain Expression Language (LCEL) to LangGraph is recommended. LCEL’s linear structure poorly supports iterative RAG workflows, where conditional logic or loops become necessary. Reserve LCEL for individual components and use LangGraph to manage the overarching flow.