OpenAI's Whisper large-v2 achieves disastrous zero-shot accuracy for Nepali financial commands.

OpenAI Whisper large-v2 demonstrates poor accuracy for Nepali financial voice command recognition tasks.

Reliability issues plague the use of OpenAI's Whisper large-v2 when processing niche, low-resource languages like Nepali, specifically regarding financial transactions. Documented in the SpeakPay paper, the zero-shot Word Error Rate (WER) hits 129.95% for Nepali financial commands, crossing the threshold for practical utility. A WER exceeding 100% signals systemic hallucinations and improper token insertions rather than simple mistakes, making the tool inadequate for precision-based financial use.

This model failure stems from zero-shot deployment, as the multilingual training within Whisper lacks specialized industry terminology. The model fails to distinguish between general conversation and domain-specific banking or wallet commands when applied to Nepali financial contexts. For individuals depending on voice-based transactions, a 129.95% WER results in misinterpreted critical commands, creating risks for operational failures in financial agents.

Researchers created NepFinSpeech-403, a dataset consisting of 403 utterances focused on core wallet operations like fund loading, balance checks, and money transfers, to mitigate these gaps. This targeted dataset acts as a bridge between general language understanding and domain-specific accuracy, establishing a benchmark for ASR performance in Nepali financial terminology.

Efficiency in adapting large models is achieved by utilizing LoRA (Low-Rank Adaptation), a technique that injects trainable rank decomposition matrices into Transformer layers rather than updating all parameters. This method minimizes computational overhead while facilitating fine-tuning with compact, specialized datasets like NepFinSpeech-403. Significant performance improvements in low-resource languages occur without requiring millions of labeled examples, proving the effectiveness of parameter-efficient fine-tuning for domain adaptation.

Developers building voice-activated financial tools in Nepal must recognize that general-purpose models fail in high-stakes environments without domain-specific refinement. Shifting from a 129.95% WER to a functional system highlights the necessity of using specialized data and techniques like LoRA for reliable production deployment. While this document presents the opinion of a single Google employee and not the entire firm, it raises interesting points as shared via SemiAnalysis. This publication remains ad-free and reader-supported. The uncomfortable truth remains that neither OpenAI nor the broader industry is currently positioned to win this arms race. While internal factions squabble, a third party has been quietly gaining ground. Many items currently considered major open problems exist in people’s hands today, such as running foundation models on a Pixel 6 at 5 tokens / sec. The modifications provided here involve only formatting and the removal of internal links.