speech enhancement sgmse

提供商sp-uhh
分类speech-enhancement
许可证mit
下载量16
星标0

简介

SGMSE 是一款基于分数生成模型（Score-based Generative Models）的语音增强工具，旨在解决复杂环境下的噪声去除问题。与传统的谱减法或简单的滤波不同，它通过生成式方法重建纯净语音，能更有效地处理非平稳噪声，显著提升语音的清晰度和自然度。对于需要处理录音素材、开发语音助手或优化远程会议音频的开发者来说，它提供了极高的音频还原质量。该模型采用 MIT 协议，开源且灵活，上手难度中等，可作为语音处理管线中的预处理模块，与 Whisper 等 ASR 识别工具配合使用以提升识别准确率。

核心亮点

  • 基于生成式模型，有效去除复杂非平稳噪声
  • 显著提升语音自然度，还原纯净人声
  • MIT 协议开源，方便集成到自有产品线
  • 可作为 ASR 预处理，大幅提高语音识别率

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("sp-uhh/speech-enhancement-sgmse")
tokenizer = AutoTokenizer.from_pretrained("sp-uhh/speech-enhancement-sgmse")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download sp-uhh/speech-enhancement-sgmse

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download sp-uhh/speech-enhancement-sgmse config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('sp-uhh/speech-enhancement-sgmse')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/sp-uhh/speech-enhancement-sgmse

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/sp-uhh/speech-enhancement-sgmse

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('sp-uhh/speech-enhancement-sgmse')
tokenizer = AutoTokenizer.from_pretrained('sp-uhh/speech-enhancement-sgmse')

完整文档

来源: HuggingFace

---
language: en
tags:

  • speech-enhancement

  • dereverberation

  • diffusion-models

  • generative-models

  • pytorch

  • audio-processing

license: mit
datasets:
  • VoiceBank-DEMAND

  • WSJ0-CHiME3

  • WSJ0-REVERB

  • EARS-WHAM

  • EARS-Reverb

model_name: speech-enhancement-dereverberation-diffusion
model_type: diffusion-based-generative-model
library_name: pytorch
key_files:
  • model.py

  • train.py

  • inference.py

inference: true
pipeline_tag: audio-to-audio
---

Speech Enhancement and Dereverberation with Diffusion-based Generative Models

<img src="https://raw.githubusercontent.com/sp-uhh/sgmse/main/diffusion_process.png" width="500" alt="Diffusion process on a spectrogram: In the forward process noise is gradually added to the clean speech spectrogram x0, while the reverse process learns to generate clean speech in an iterative fashion starting from the corrupted signal xT.">

This repository contains the official PyTorch implementations for the papers:

Audio examples and supplementary materials are available on our SGMSE project page, EARS project page, and Investigating training objectives project page.

Key Files

The following files are essential for this model:

  • model.py: Defines the model architecture and training processes.
  • train.py: Script for training the model.
  • enhancement.py: Used for evaluating and enhancing audio files.
  • requirements.txt: Lists the necessary dependencies.
  • calc_metrics.py: Script for calculating evaluation metrics.

Follow-up work

Please also check out our follow-up work with code available:

Installation

  • Create a new virtual environment with Python 3.11 (we have not tested other Python versions, but they may work).
  • Install the package dependencies via pip install -r requirements.txt.
- Let pip resolve the dependencies for you. If you encounter any issues, please check requirements_version.txt for the exact versions we used.
  • If using W&B logging (default):
- Set up a wandb.ai account - Log in via wandb login before running our code.
  • If not using W&B logging:
- Pass the option --nolog to train.py. - Your logs will be stored as local CSVLogger logs in lightning_logs/.

Pretrained checkpoints

  • For the speech enhancement task, we offer pretrained checkpoints for models that have been trained on the VoiceBank-DEMAND and WSJ0-CHiME3 datasets, as described in our journal paper [2]. You can download them here.
- SGMSE+ trained on VoiceBank-DEMAND: gdown 1_H3EXvhcYBhOZ9QNUcD5VZHc6ktrRbwQ - SGMSE+ trained on WSJ0-CHiME3: gdown 16K4DUdpmLhDNC7pJhBBc08pkSIn_yMPi
  • For the dereverberation task, we offer a checkpoint trained on our WSJ0-REVERB dataset. You can download it here.
- SGMSE+ trained on WSJ0-REVERB: gdown 1eiOy0VjHh9V9ZUFTxu1Pq2w19izl9ejD - Note that this checkpoint works better with sampler settings --N 50 --snr 0.33.
  • For 48 kHz models [3], we offer pretrained checkpoints for speech enhancement, trained on the EARS-WHAM dataset, and for dereverberation, trained on the EARS-Reverb dataset. You can download them here.
- SGMSE+ trained on EARS-WHAM: gdown 1t_DLLk8iPH6nj8M5wGeOP3jFPaz3i7K5 - SGMSE+ trained on EARS-Reverb: gdown 1PunXuLbuyGkknQCn_y-RCV2dTZBhyE3V
  • For the investigating training objectives checkpoints [4], we offer the pretrained checkpoints here
- M1: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m1.ckpt - M2: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m2.ckpt - M3: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m3.ckpt - M4: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m4.ckpt - M5: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m5.ckpt - M6: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m6.ckpt - M7: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m7.ckpt - M8: wget https://www2.informatik.uni-hamburg.de/sp/audio/publications/icassp2025_gense/checkpoints/m8.ckpt

Usage:

  • For resuming training, you can use the --ckpt option of train.py.

  • For evaluating these checkpoints, use the --ckpt option of enhancement.py (see section Evaluation below).

Training

Training is done by executing train.py. A minimal running example with default settings (as in our paper [2]) can be run with

bash
python train.py --base_dir <your_base_dir>

where your_base_dir should be a path to a folder containing subdirectories train/ and valid/ (optionally test/ as well). Each subdirectory must itself have two subdirectories clean/ and noisy/, with the same filenames present in both. We currently only support training with .wav files.

To see all available training options, run python train.py --help. Note that the available options for the SDE and the backbone network change depending on which SDE and backbone you use. These can be set through the --sde and --backbone options.

Note:

  • Our journal [2] uses --backbone ncsnpp.

  • For the 48 kHz model [3], use --backbone ncsnpp_48k --n_fft 1534 --hop_length 384 --spec_factor 0.065 --spec_abs_exponent 0.667 --sigma-min 0.1 --sigma-max 1.0 --theta 2.0

  • Our Interspeech paper [1] uses --backbone dcunet. You need to pass --n_fft 512 to make it work.

- Also note that the default

查看官方来源