Optimizing LoRA Hyperparameters for Fine-Tuning Llama 3 on Domain-Specific Datasets

Stop blindly following the "default" LoRA settings from Hugging Face tutorials; if you're fine-tuning Llama 3 on niche domain data (medical, legal, or proprietary codebase), the standard r=8 and alpha=16 usually result in a model that either forgets its base knowledge or fails to capture the specific terminology of your dataset.

In my recent attempts to bake a specialized technical manual into Llama 3, I found that the relationship between rank (r) and alpha (alpha) is where most people trip up. The general rule of thumb—setting alpha to 2x the rank—is a starting point, but for domain-specific adaptation, you often need a higher rank to capture the complexity of new jargon.

I've had the best results pushing r to 32 or 64. However, the "gotcha" here is the learning rate. If you increase the rank without lowering the learning rate, the model tends to diverge or collapse into repetitive patterns. For Llama 3, I've found a sweet spot with learning_rate=2e-4 and a cosine scheduler, but only when using a very small batch size.

Here is the specific config I'm currently using with PEFT and Unsloth (which is significantly faster for Llama 3):

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=64, 
    lora_alpha=128, 
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], 
    lora_dropout=0.05, 
    bias="none", 
    task_type="CAUSAL_LM"
)

The most critical change I made was targeting all linear layers. Most basic scripts only target q_proj and v_proj. If you want the model to actually learn new domain knowledge rather than just changing its "tone," you must include the MLP layers (gate_proj, up_proj, down_proj). This increases the VRAM footprint slightly, but the gain in factual accuracy on domain-specific queries is massive.

Another productivity tip: use a validation set that specifically tests for "catastrophic forgetting." I keep a small set of general knowledge questions. If the model starts failing those while improving on the domain data, your lora_alpha is likely too high, or you're overtraining.

My current hyperparameter checklist for domain tuning:

• Rank (r): 32 to 64 for complex domains; 8 to 16 for simple style transfer.
• Alpha: Always keep it proportional to r (usually 2x), but if the model is "under-learning," bump it up slightly.
• Target Modules: Always use all linear layers for domain-specific tasks.
• Epochs: 1 to 3. Llama 3 overfits incredibly fast on small, high-quality datasets.
• Weight Decay: 0.01 to prevent the weights from exploding during the early stages of training.

If you are running this on a single A100 or 3090/4090, make sure you're using 4-bit quantization (QLoRA). The loss in precision is negligible compared to the ability to fit a larger rank into memory. For those using Cursor to write their training scripts, I recommend prompting it specifically to use the SFTTrainer from the trl library, as it handles the padding and packing much more efficiently than a raw PyTorch loop.