Optimizing Ollama Performance for Local RAG using Quantized Llama 3 Models

Running Llama 3 locally via Ollama for RAG (Retrieval-Augmented Generation) usually hits a wall where the context window eats your VRAM, causing the generation speed to plummet from "instant" to "one word per second." Most people just download the default 8B model, but if you're actually building a RAG pipeline, you need to be surgical about which quantization level you use to balance retrieval accuracy with inference speed.

The default llama3 tag in Ollama is usually a 4-bit quantization (q4_0). For basic chat, it's fine. But for RAG, where the model has to parse a large chunk of retrieved context before answering, q4 can sometimes "hallucinate" details from the provided text because the weights are too compressed to maintain precise attention over long prompts.

I've found that switching to q6_K or q8_0 is the sweet spot for local RAG. The jump from 4-bit to 6-bit is barely noticeable in VRAM usage on a 12GB or 16GB card, but the coherence when summarizing three or four retrieved documents improves significantly.

Here is how I actually set up my Ollama environment to prioritize performance for a RAG workflow:

1. Pulling the specific quantization
Don't rely on the generic tag. Pull the specific K-quants to ensure you aren't wasting memory or losing too much precision.

ollama run llama3:8b-instruct-q6_K

2. Tuning the Modelfile for RAG
The biggest productivity gain comes from creating a custom Modelfile. By default, Ollama's context window (num_ctx) is often too small for RAG, leading to the model "forgetting" the beginning of the retrieved context. I always bump this up and tweak the temperature to keep the model grounded in the facts.

Create a file named rag-llama3.Modelfile:

FROM llama3:8b-instruct-q6_K

# Increase context window to 8k or 16k depending on your VRAM
PARAMETER num_ctx 8192
# Lower temperature to reduce hallucinations in RAG
PARAMETER temperature 0.2
# Set a system prompt that forces the model to use the context
SYSTEM """
You are a technical assistant. Use the provided context to answer the question. 
If the answer isn't in the context, say you don't know. 
Do not use outside knowledge.
"""

Then build it:

ollama create rag-llama3 -f rag-llama3.Modelfile

3. Handling the "Context Bloat" Gotcha
One thing that tripped me up early on was the interaction between the embedding model and the LLM. If you use a heavy embedding model (like bge-large) and a high-quantization Llama 3, you might trigger swap memory on your GPU, which kills performance.

My optimized stack for a 12GB GPU:

• LLM: Llama 3 8B (q6_K) via Ollama.
• Embeddings: bge-small-en-v1.5 (Run via Sentence-Transformers in Python, not Ollama, to keep the GPU memory overhead predictable).
• Vector Store: Qdrant (Local Docker container).

4. Performance Benchmarks
On my RTX 3060 (12GB), the difference in tokens per second (t/s) is marginal, but the "Time to First Token" (TTFT) spikes if num_ctx is set too high. I've found that 8192 is the limit before I see a noticeable lag. If you're pushing 16k context, you absolutely must drop back to q4_K_M to avoid hitting system RAM.

If you're still seeing slow responses, check your OLLAMA_NUM_PARALLEL environment variable. By default, Ollama handles requests sequentially. If your RAG app sends multiple queries (like for query expansion or re-ranking), setting OLLAMA_NUM_PARALLEL=2 can help, provided you have the VRAM to support multiple KV caches.