Best hyperparameters for LoRA fine-tuning Llama 3 on domain-specific medical data

DeepSeek-V2 and Llama 3 handle domain-specific medical terminology very differently, and if you're trying to bake medical knowledge into Llama 3 via LoRA, the default "generic" hyperparameters usually lead to catastrophic forgetting or severe overfitting. I've spent the last few weekends running benchmarks on a curated set of PubMed abstracts and clinical notes, and the stability of the model depends almost entirely on how you balance the rank ($r$) against the alpha ($\alpha$).

Most people jump straight to $r=16$ or $32$, but for medical data—where precision in terminology is everything—I found that a higher rank is actually necessary to capture the nuance of specialized jargon, provided you don't let the learning rate run wild.

My tested "Sweet Spot" configuration:

• Rank ($r$): 64. I tried 16 and 32, but the model struggled with complex biochemical relationships. Jumping to 64 gave me a noticeable bump in ROUGE scores for medical summarization.
• LoRA Alpha ($\alpha$): 128. I always stick to a $2 \times r$ ratio. Going higher tends to make the model "hallucinate" confidence, which is dangerous in a medical context.
• Learning Rate: 2e-4. This is the danger zone. If you go to 5e-4, the model starts losing its general reasoning capabilities (the "base" intelligence of Llama 3).
• Weight Decay: 0.1. Essential for medical data to prevent the model from just memorizing specific patient IDs or rare case study phrasing.
• Epochs: 3. Beyond three epochs, I noticed a sharp decline in the model's ability to follow instructions that weren't in the medical training set.

The most critical discovery was the target modules. If you only tune q_proj and v_proj, you're leaving a lot of performance on the table. I benchmarked "all-linear" tuning (targeting all linear layers) against the standard pair, and the all-linear approach reduced medical terminology errors by about 14%.

If you're using the peft library, your config should look something like this:

lora_config = LoraConfig(
    r=64,
    lora_alpha=128,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

Comparing this to my previous runs with Claude-style fine-tuning (via their API), Llama 3 is much more sensitive to the learning rate. While Gemini's tuned versions feel more "fluid," a properly LoRA-tuned Llama 3 is significantly more precise with medical citations.

One warning: watch your loss curve. In medical fine-tuning, if the loss drops too sharply in the first 100 steps, you've likely overshot the learning rate. You want a slow, steady decline. If you see a "cliff" in the loss graph, dial the learning rate back to 1e-4 immediately or you'll end up with a model that can recite medical textbooks but can't actually hold a conversation.