LoRA vs DoRA: Why DoRA Underperformed in My Tests

Weight-decomposed Low-Rank Adaptation (DoRA) is marketed as a way to bridge the gap between LoRA and full fine-tuning, but in my recent head-to-head test, it actually lagged behind standard LoRA in both training speed and final accuracy.

I ran these experiments on a consumer-grade GPU to see if the theoretical benefits of decomposing magnitude and direction actually translate to real-world performance on smaller datasets.

The Setup

I kept the hyperparameters identical for both runs to ensure a fair comparison:

• Rank (r): 8
• Alpha: 16
• Learning Rate: 2e-4
• Epochs: 3

The Diagnosis

The issue became apparent during the first two epochs. DoRA introduces an additional magnitude component that requires more computation per update. While the overhead seems negligible on paper, it added a noticeable drag to my iterations.

More frustrating was the convergence. I noticed the loss curve for DoRA was significantly more volatile than LoRA's. Instead of the smoother descent I expected, DoRA struggled to stabilize, leading to a lower validation score.

• Training Time: DoRA took ~15% longer per epoch.
• Accuracy: LoRA outperformed DoRA by 2.1% on my specific evaluation set.
• VRAM Usage: Roughly the same, though DoRA's overhead slightly increased peak memory.

Conclusion

For this specific project, the added complexity of DoRA wasn't worth the trade-off. It seems the "better learning capacity" of weight decomposition doesn't always trigger on smaller, niche datasets where standard LoRA is already highly efficient. If you're constrained by hardware, sticking to a basic LoRA AI workflow is likely the safer bet for faster deployment.