Optimizing Llama 3 8B for RAG using Unsloth and LoRA adapters

DeepSeek-V2 is great, but for local RAG pipelines where latency is king, Llama 3 8B is still the sweet spot if you know how to squeeze it. I’ve spent the last few days running head-to-head tests between a vanilla Llama 3 8B and a version fine-tuned via Unsloth using LoRA adapters specifically on a domain-specific corpus (technical documentation for a proprietary API), and the difference in "hallucination rate" is stark.

The biggest bottleneck with small models in RAG isn't usually the retrieval—it's the model's tendency to ignore the provided context in favor of its pre-trained weights when the prompt gets slightly complex. By using Unsloth, I managed to cut the VRAM requirement for training down by about 60% compared to standard Hugging Face PEFT, which lets you push the context window further without hitting OOM on a consumer 3090/4090.

Here is the performance breakdown based on my benchmarks:

Vanilla Llama 3 8B (Base)

• Context Adherence: Moderate. Often drifts into general knowledge if the retrieved chunk is dense.
• Inference Speed: Blazing fast, but requires heavy prompting (few-shot) to stay on track.
• Accuracy: High on general facts, but fails on niche technical terminology in the RAG chunks.

Unsloth LoRA-tuned Llama 3 8B

• Context Adherence: Very High. The adapter effectively "teaches" the model to prioritize the Context: block over its internal weights.
• Inference Speed: Negligible overhead. Since the LoRA weights are merged or handled efficiently, I'm seeing nearly the same tokens/sec as the base model.
• Accuracy: Significant jump in precision for domain-specific queries.

The key to making this work is the dataset formatting. If you just feed it raw text, you're wasting your time. I found that using a "Context-Question-Answer" triplet where the answer is strictly derived from the context is the only way to kill the hallucinations.

I used a basic training loop in a Jupyter notebook. If you're setting this up, make sure you're using the 4-bit quantized version to keep the memory footprint low:

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/llama-3-8b-bnb-4bit",
    max_seq_length = 2048,
    load_in_4bit = True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r = 16, 
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,
    lora_dropout = 0, 
    bias = "none",    
)

Compared to Claude 3.5 Sonnet, which is obviously superior in reasoning, this optimized 8B model is surprisingly competitive for narrow, closed-domain RAG. The latency is the real winner; I'm getting responses in milliseconds that are 90% as accurate as the frontier models for this specific use case.

The trade-off is the "catastrophic forgetting" risk. If you overfit the LoRA adapter to your RAG data, the model starts losing its general conversational ability. I found that a low rank (r=16) and a very low learning rate are essential to keep the model from becoming a one-trick pony. If you go up to r=64, it starts ignoring the user's intent and just regurgitates the documentation verbatim, which isn't what you want for a natural UI.