Name: mpse smoe speech enhancement
Author: AdityaRaikar

Overview

MPSE SMOE is a specialized speech enhancement model designed to isolate clean vocals from noisy environments. Unlike general-purpose noise cancellation, this architecture leverages a Sparsely Gated Mixture-of-Experts (SMOE) approach, allowing it to handle diverse acoustic interference patterns without the computational overhead of a dense model. For developers, this means a more efficient deployment cycle for real-time audio processing pipelines. It is particularly suited for VoIP applications, podcast post-production tools, and accessibility software where voice clarity is critical. Integration is straightforward via standard audio tensors, and since it is released under the MIT license, it offers maximum flexibility for commercial integration without restrictive licensing hurdles.

Highlights

SMOE architecture reduces computational load during inference
Highly effective at removing complex background noise
Permissive MIT license for easy commercial deployment
Optimized for real-time speech restoration pipelines

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("AdityaRaikar/mpse-smoe-speech-enhancement")
tokenizer = AutoTokenizer.from_pretrained("AdityaRaikar/mpse-smoe-speech-enhancement")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download AdityaRaikar/mpse-smoe-speech-enhancement

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download AdityaRaikar/mpse-smoe-speech-enhancement config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('AdityaRaikar/mpse-smoe-speech-enhancement')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/AdityaRaikar/mpse-smoe-speech-enhancement

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/AdityaRaikar/mpse-smoe-speech-enhancement

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('AdityaRaikar/mpse-smoe-speech-enhancement')
tokenizer = AutoTokenizer.from_pretrained('AdityaRaikar/mpse-smoe-speech-enhancement')

Full Documentation

来源: HuggingFace

---
tags:
- speech-enhancement
- audio-to-audio
- MP-SENet
- supervised-mixture-of-experts
- S-MoE
- dual-bandwidth
- narrowband
- wideband
license: mit
---

MP-SENet + Supervised Mixture of Experts (S-MoE) for Dual-Bandwidth Speech Enhancement

Speech enhancement model handling both Narrowband (NB, 8kHz) and Wideband (WB, 16kHz) audio using Supervised Mixture of Experts.

Based on:

MP-SENet — STFT-domain magnitude-phase speech enhancement backbone

S-MoE — Supervised MoE with deterministic hard gating by bandwidth metadata

Pre-trained Baseline

The official pre-trained MP-SENet baseline (from yxlu-0102/MP-SENet) is included at pretrained/g_best_vb.pt:

Trained on VoiceBank+DEMAND, 16kHz, 100 epochs

2.263M params, 247 state dict keys

No need to train baseline from scratch — S-MoE training initializes directly from this checkpoint

Training (S-MoE only)

bash

pip install -r requirements-gpu.txt
S-MoE training (downloads pre-trained baseline automatically)
SMOE_EPOCHS=60 BATCH_SIZE=2 GRAD_ACCUM_STEPS=2 python train.py

The script automatically:
1. Downloads VoiceBank+DEMAND (~5GB)
2. Pre-extracts WB/NB data into separate folders (no on-the-fly resampling)
3. Downloads official pre-trained MP-SENet baseline from this repo
4. Initializes S-MoE experts from baseline (shared → both WB+NB experts)
5. Trains S-MoE on joint WB+NB data for 60 epochs
6. Pushes trained model to HF Hub

Data Pipeline

All NB/WB data is pre-extracted to separate folders before training:

| Folder | Type | Processing | bandwidth_id |
|---|---|---|---|
| original_wb/ | WB | Original 16kHz (symlinks) | 0 |
| simple_nb/ | NB | 16k→8k→16k resampled | 1 |
| codec_nb/ | NB | G.711 A-law/mu-law (round-robin) | 1 |

Architecture

| Component | Baseline | S-MoE |
|---|---|---|
| Total params | 2.263M | 3.587M |
| Active params (inference) | 2.263M | 2.263M (same!) |
| FFN type | Single GRU-FFN | 2× GRU-FFN experts, hard gated by bandwidth_id |
| Attention | Shared | Shared (unchanged) |

Environment Variables

| Variable | Default | Description |
|---|---|---|
| SMOE_EPOCHS | 60 | Epochs for S-MoE training |
| BATCH_SIZE | 2 | GPU micro-batch size |
| GRAD_ACCUM_STEPS | 2 | Gradient accumulation (effective BS = BATCH_SIZE × GRAD_ACCUM_STEPS) |
| PESQ_EVERY_N | 10 | Compute PESQ for discriminator every N steps (expensive) |
| LR | 5e-4 | Learning rate |
| EVAL_INTERVAL | 5 | Evaluate every N epochs |
| BASELINE_REPO | AdityaRaikar/mpse-smoe-speech-enhancement | HF repo with pre-trained baseline |
| BASELINE_FILE | pretrained/g_best_vb.pt | Checkpoint filename in repo |

Files

| File | Description |
|---|---|
| train.py | Combined data prep + S-MoE training (self-contained) |
| pretrained/g_best_vb.pt | Official pre-trained MP-SENet baseline (VoiceBank+DEMAND) |
| smoe_models.py | Model definitions with detailed docstrings |
| data_preparation.py | Standalone NB/WB codec pipeline |
| config.json | Default hyperparameters |

References

bibtex

@article{lu2023mpse,
  title={MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra},
  author={Lu, Ye-Xin and Ai, Yang and Ling, Zhen-Hua},
  journal={arXiv preprint arXiv:2305.13686},
  year={2023}
}