Optimizing Llama 3 70B for long-context RAG using QLoRA fine-tuning

Llama 3 70B is a beast for general reasoning, but out of the box, it tends to suffer from "lost in the middle" syndrome when you push the context window for RAG, especially if your retrieved chunks are noisy. I spent the last few days running a series of QLoRA experiments to see if fine-tuning on domain-specific long-context pairs could actually stabilize the needle-in-a-haystack retrieval performance.

The setup involved 4x A100s (80GB) using Unsloth to keep the memory overhead low. If you're trying to do this on consumer gear, you're going to need heavy 4-bit quantization because the base 70B is too bloated for standard LoRA on 24GB cards.

My core finding is that vanilla Llama 3 70B often hallucinates when the answer is buried in a 32k+ token window, even if the retrieval system (like BGE-M3) successfully puts the right chunk in the prompt. By applying QLoRA specifically to the attention layers and the MLP blocks with a focus on long-form synthetic Q&A pairs, the "precision" of the retrieval-augmented generation improved significantly.

Here is how the performance broke down compared to GPT-4o and Claude 3.5 Sonnet on a 20k token context test:

Llama 3 70B (Base)

• Accuracy: Frequent failure to cite specific page numbers or IDs from the context.
• Latency: Fast, but prone to looping when the context is too dense.
• Verdict: Unreliable for high-stakes RAG where precision is non-negotiable.

Llama 3 70B (QLoRA Tuned)

• Accuracy: Massive jump in "grounding." It stopped making up facts and started sticking to the provided context.
• Latency: Negligible difference from the base model since the adapters are lightweight.
• Verdict: Now rivals Claude 3.5 in narrow domain retrieval, though still lacks Claude's general nuance.

Claude 3.5 Sonnet

• Accuracy: Gold standard for long-context. Almost zero "lost in the middle" issues.
• Latency: Slower than the local 70B.
• Verdict: The benchmark to beat, but expensive for high-throughput pipelines.

To get this working, I used a specific rank for the LoRA adapters. Setting r=64 and alpha=128 seemed to be the sweet spot. Anything lower and the model didn't pick up the long-context patterns; anything higher and I started seeing catastrophic forgetting of general knowledge.

For those wanting to replicate this, here is the configuration snippet I used for the SFT trainer:

training_args = TrainingArguments(
    per_device_train_batch_size = 2,
    gradient_accumulation_steps = 4,
    warmup_steps = 10,
    max_steps = 100, 
    learning_rate = 2e-4,
    fp16 = not torch.cuda.is_bf16_supported(),
    bf16 = torch.cuda.is_bf16_supported(),
    logging_steps = 1,
    optim = "adamw_8bit",
    weight_decay = 0.01,
    lr_scheduler_type = "linear",
    seed = 3407,
)

One major caveat: the data quality for the fine-tuning set is everything. I used a "Context-Question-Answer" triplet where the answer explicitly required synthesizing information from two different parts of the 16k-32k token window. If you just feed it short snippets, the model won't learn to attend to the full window.

If you're deciding between just increasing the context window via Rope scaling or actually doing QLoRA, go with the tuning. Scaling the window just gives the model more "room" to be confused; QLoRA actually teaches it how to use that room.