Optimizing Qwen2.5-Coder for Local RAG Workflows using Ollama and LangChain

Qwen2.5-Coder-7B is arguably the best "bang-for-buck" local model right now for RAG because it doesn't hallucinate syntax as much as Llama 3.1 when dealing with technical documentation. I've been using it to build a local codebase indexer, and the trick to making it actually usable—rather than just a slow chat bot—is strictly controlling the context window and the prompt template.

If you just run ollama run qwen2.5-coder, you're leaving performance on the table. The model can struggle with "lost in the middle" syndrome if your retrieved chunks are too long or poorly formatted.

I found that creating a custom Modelfile is non-negotiable for RAG. You need to force the model into a "concise assistant" mode so it stops explaining what a variable is and just answers the question based on the provided context.

# Custom Modelfile for Qwen2.5-Coder RAG
FROM qwen2.5-coder:7b
PARAMETER temperature 0.2
PARAMETER num_ctx 8192
SYSTEM """
You are a technical expert. Use the provided context to answer the query. 
If the answer isn't in the context, say 'Context insufficient'. 
Keep answers brief and use Markdown for code.
"""

Run ollama create qwen-rag -f Modelfile to bake this in. Lowering the temperature to 0.2 is critical; otherwise, the model tries to be "creative" with your API calls, which is a nightmare for debugging.

On the LangChain side, the biggest bottleneck is usually the retrieval quality, not the LLM. I switched from simple RecursiveCharacterTextSplitter to a semantic-aware approach. Qwen2.5-Coder handles structured data well, so I wrap my retrieved documents in XML-style tags. It helps the model distinguish between the "knowledge" and the "question."

Here is the implementation pattern I'm using to feed the context into the chain:

from langchain_community.llms import Ollama
from langchain_core.prompts import ChatPromptTemplate

llm = Ollama(model="qwen-rag")

# Use XML tags to delineate context—Qwen responds much better to this
template = """<context>
{context}
</context>

Question: {question}
Answer:"""

prompt = ChatPromptTemplate.from_template(template)
chain = prompt | llm

# Example invocation
response = chain.invoke({
    "context": "The UserAuth class handles JWT validation in auth.py",
    "question": "Where is JWT validation handled?"
})

One major "gotcha" I hit: if you're using a GPU with limited VRAM (like 8GB or 12GB), Ollama might offload some layers to the CPU, causing a massive latency spike during the "generation" phase of RAG. I solved this by explicitly setting num_gpu in the Modelfile or ensuring no other heavy electron apps are eating VRAM.

Productivity Gains:

• Indexing Speed: Using FastEmbed instead of full SentenceTransformers cut my embedding time by 60%.
• Accuracy: Moving from plain text prompts to XML-tagged context reduced "hallucinated functions" by nearly 30% in my tests.
• Latency: The custom Modelfile with num_ctx 8192 (instead of the default) keeps the KV cache manageable while still fitting 4-5 decent-sized code snippets.

If you're seeing the model ramble, check your stop sequences. Adding stop=["</context>", "\nQuestion:"] to your Ollama options prevents the model from trying to simulate a conversation with itself.