Diffusion-gemma-asr: 15x Faster Than Whisper

Imagine a world where you don't have to wait an eternity for your AI to transcribe a voice memo. While we've all been treating Whisper as the gold standard, diffusion-gemma-asr just crashed the party by being 15x faster. The trick? It stops playing the "one token at a time" game and just denoises the entire transcript in parallel. It's basically the difference between typing a letter character by character and just having the whole page appear out of a fog.

The technical flex here is actually hilarious: they froze almost everything. They used a frozen Whisper-small encoder and frozen DiffusionGemma weights, then just slapped on a tiny 42M parameter adapter (which is like 0.16% of the model). They used a convolutional projector to squash 1,500 audio frames into 188 tokens, and some LoRA adapters to make sure the model didn't just ignore the audio entirely.

Here is the high-level AI workflow they used for the training:

1. The Scaffold: They used CTC loss as a "training scaffold" to force the projector to actually learn something before the attention layers decided it was useless.
2. The Process: DiffusionGemma doesn't use mask tokens; it starts with a canvas of random vocab tokens and iteratively replaces the "uncertain" ones until the text settles.
3. The Result: After training on Librispeech and fine-tuning on FLEURS and VoxPopuli, they hit a 6.6% WER on Librispeech.

Sure, it's still not beating the massive Whisper models because those were trained on millions of hours of audio while this one only saw about 219 hours. But the fact that a frozen diffusion LLM can pick up a whole new modality with a tiny adapter is a massive win for anyone trying to build lean, fast LLM agents.

If you want to poke around the implementation, the adapter weights and inference scripts are open source.

https://huggingface.co/spaces/interfaze-ai/diffusion-gemma-asr-demo