speech enhancement sgmse

Providersp-uhh
Categoryspeech-enhancement
Licensemit
Downloads16
Stars0

Overview

SGMSE is a generative speech enhancement model designed to isolate clean voice signals from noisy environments. Unlike traditional spectral subtraction methods, it leverages score-based generative modeling to reconstruct high-fidelity audio, effectively handling non-stationary noise and reverberation. For developers, this means better performance in real-world scenarios like VoIP, podcast post-production, or accessibility tools where signal-to-noise ratios are unpredictable. It integrates easily into Python-based audio pipelines and, being MIT-licensed, offers full flexibility for commercial deployment without restrictive licensing overhead. Compared to standard CNN-based denoisers, SGMSE prioritizes natural sound quality and reduced artifacts, though it requires careful consideration of inference latency depending on the hardware target.

Highlights

  • Generative score-based approach for superior noise suppression
  • MIT license allows seamless commercial integration
  • Effective reconstruction of high-fidelity speech in noisy environments
  • Reduced audio artifacts compared to traditional denoisers

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("sp-uhh/speech-enhancement-sgmse")
tokenizer = AutoTokenizer.from_pretrained("sp-uhh/speech-enhancement-sgmse")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance 
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository 
huggingface-cli download sp-uhh/speech-enhancement-sgmse

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir) 
huggingface-cli download sp-uhh/speech-enhancement-sgmse config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('sp-uhh/speech-enhancement-sgmse')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://huggingface.co/sp-uhh/speech-enhancement-sgmse

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/sp-uhh/speech-enhancement-sgmse

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers 
pip install -U transformers torch

Load the model and run inference

Load the model and run inference 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('sp-uhh/speech-enhancement-sgmse')
tokenizer = AutoTokenizer.from_pretrained('sp-uhh/speech-enhancement-sgmse')

Full Documentation

来源: HuggingFace

---
language: en
tags:

  • speech-enhancement

  • dereverberation

  • diffusion-models

  • generative-models

  • pytorch

  • audio-processing

license: mit
datasets:
  • VoiceBank-DEMAND

  • WSJ0-CHiME3

  • WSJ0-REVERB

  • EARS-WHAM

  • EARS-Reverb

model_name: speech-enhancement-dereverberation-diffusion
model_type: diffusion-based-generative-model
library_name: pytorch
key_files:
  • model.py

  • train.py

  • inference.py

inference: true
pipeline_tag: audio-to-audio
---

Speech Enhancement and Dereverberation with Diffusion-based Generative Models

<img src="https://raw.githubusercontent.com/sp-uhh/sgmse/main/diffusion_process.png" width="500" alt="Diffusion process on a spectrogram: In the forward process noise is gradually added to the clean speech spectrogram x0, while the reverse process learns to generate clean speech in an iterative fashion starting from the corrupted signal xT.">

This repository contains the official PyTorch implementations for the papers:

Audio examples and supplementary materials are available on our SGMSE project page, EARS project page, and Investigating training objectives project page.

Key Files

The following files are essential for this model:

  • model.py: Defines the model architecture and training processes.
  • train.py: Script for training the model.
  • enhancement.py: Used for evaluating and enhancing audio files.
  • requirements.txt: Lists the necessary dependencies.
  • calc_metrics.py: Script for calculating evaluation metrics.

Follow-up work

Please also check out our follow-up work with code available:

Installation

  • Create a new virtual environment with Python 3.11 (we have not tested other Python versions, but they may work).
  • Install the package dependencies via pip install -r requirements.txt.
- Let pip resolve the dependencies for you. If you encounter any issues, please check requirements_version.txt for the exact versions we used.
  • If using W&B logging (default):
- Set up a wandb.ai account - Log in via wandb login before running our code.
  • If not using W&B logging:
- Pass the option --nolog to train.py. - Your logs will be stored as local CSVLogger logs in lightning_logs/.

Pretrained checkpoints

  • For the speech enhancement task, we offer pretrained checkpoints for models that have been trained on the VoiceBank-DEMAND and WSJ0-CHiME3 datasets, as described in our journal paper [2]. You can download them here.
- SGMSE+ trained on VoiceBank-DEMAND: gdown 1_H3EXvhcYBhOZ9QNUcD5VZHc6ktrRbwQ - SGMSE+ trained on WSJ0-CHiME3: gdown 16K4DUdpmLhDNC7pJhBBc08pkSIn_yMPi
  • For the dereverberation task, we offer a checkpoint trained on our WSJ0-REVERB dataset. You can download it here.
- SGMSE+ trained on WSJ0-REVERB: gdown 1eiOy0VjHh9V9ZUFTxu1Pq2w19izl9ejD - Note that this checkpoint works better with sampler settings --N 50 --snr 0.33.
  • For 48 kHz models [3], we offer pretrained checkpoints for speech enhancement, trained on the EARS-WHAM dataset, and for dereverberation, trained on the EARS-Reverb dataset. You can download them here.
- SGMSE+ trained on EARS-WHAM: gdown 1t_DLLk8iPH6nj8M5wGeOP3jFPaz3i7K5 - SGMSE+ trained on EARS-Reverb: gdown 1PunXuLbuyGkknQCn_y-RCV2dTZBhyE3V
  • For the investigating training objectives checkpoints [4], we offer the pretrained checkpoints here
- M1: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m1.ckpt - M2: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m2.ckpt - M3: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m3.ckpt - M4: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m4.ckpt - M5: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m5.ckpt - M6: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m6.ckpt - M7: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m7.ckpt - M8: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m8.ckpt

Usage:

  • For resuming training, you can use the --ckpt option of train.py.

  • For evaluating these checkpoints, use the --ckpt option of enhancement.py (see section Evaluation below).

Training

Training is done by executing train.py. A minimal running example with default settings (as in our paper [2]) can be run with

bash
python train.py --base_dir <your_base_dir>

where your_base_dir should be a path to a folder containing subdirectories train/ and valid/ (optionally test/ as well). Each subdirectory must itself have two subdirectories clean/ and noisy/, with the same filenames present in both. We currently only support training with .wav files.

To see all available training options, run python train.py --help. Note that the available options for the SDE and the backbone network change depending on which SDE and backbone you use. These can be set through the --sde and --backbone options.

Note:

  • Our journal [2] uses --backbone ncsnpp.

  • For the 48 kHz model [3], use --backbone ncsnpp_48k --n_fft 1534 --hop_length 384 --spec_factor 0.065 --spec_abs_exponent 0.667 --sigma-min 0.1 --sigma-max 1.0 --theta 2.0

  • Our Interspeech paper [1] uses --backbone dcunet. You need to pass --n_fft 512 to make it work.

- Also note that the default

View official source
Join our Telegram