Whisper's zero-shot performance on Nepali financial commands is
The researchers tackled this by creating a specialized dataset called NepFinSpeech-403, which consists of 403 utterances covering core wallet operations like sending money, loading funds, and checking balances. What makes this interesting for anyone working on LLM agent deployment or specialized ASR (Automatic Speech Recognition) is how they used LoRA (Low-Rank Adaptation) to fine-tune the model without needing a massive compute cluster.
The impact of domain-adaptive fine-tuning
The jump in performance after applying LoRA was massive, but the most telling metric isn't actually the WER. While they managed to bring the WER down from ~130% to 42.58%, the real win is in the "Transaction Success Rate."
- Zero-shot Success Rate: 1.67%
- LoRA Fine-tuned Success Rate: 33.33%
- Relative improvement: ~20x gain in actual usability
- Devanagari numeral accuracy: Jumped from 0.0% to 73.9%
The study shows that for low-resource languages, general-purpose models are almost useless for high-precision tasks like banking. Even with a 42% error rate, the model went from being a complete brick to actually being able to facilitate transactions.
Data efficiency and the "plateau" effect
One of the most practical takeaways for developers building custom AI workflows is how much data you actually need. You don't need thousands of hours of audio to see a massive shift.
1. The 100-utterance threshold: Just 100 domain-specific utterances were enough to cut the zero-shot WER in half.
2. The 300-utterance plateau: Performance gains started to level off once they hit roughly 300 examples.
This is a huge hint for anyone trying to implement a practical tutorial for domain adaptation on a budget. If you're working with a niche dialect or a specific industry jargon, you can get significant results with a very small, high-quality dataset.
Why it still fails
Even with the fine-tuning, the model isn't perfect. The error analysis pointed to systematic issues with how the model handles numbers. Specifically, they saw "prefix hallucination" and frequent zero insertions or deletions. In a financial context, if the model hallucinates a digit or misses a zero, the entire transaction is ruined. This is a classic example of why ASR for fintech requires more than just a good language model—it requires strict post-processing or constrained decoding to ensure numerical integrity.
The researchers have made everything available, including the dataset and the model weights, which is great for anyone wanting to run a hands-on guide on fine-tuning Whisper for other low-resource languages.
https://github.com/subedibiraj/speakpay