fastspeech2 conformer with hifigan

提供商espnet
分类text-to-audio
许可证apache-2.0
下载量8.8K
星标0

简介

这是一个由 ESPnet 提供的端到端语音合成方案，结合了 FastSpeech 2 的非自回归架构、Conformer 的上下文建模能力以及 HiFi-GAN 的高保真声码器。相比传统的 TTS 模型，它在保证语音自然度和韵律感的同时，极大地提升了推理速度，解决了合成语音机械感强的问题。对于开发者而言，该模型非常适合需要实时性要求较高且对音质有一定追求的场景，如智能助手、有声书阅读或虚拟人配音。由于基于成熟的开源框架，上手难度中等，是构建高质量文本转语音流水线的理想选择。

核心亮点

  • 非自回归架构，实现极速语音合成推理
  • Conformer 增强语调自然度，告别机械音
  • HiFi-GAN 声码器确保输出音频高保真
  • 适配 ESPnet 生态，适合工业级 TTS 部署

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("espnet/fastspeech2_conformer_with_hifigan")
tokenizer = AutoTokenizer.from_pretrained("espnet/fastspeech2_conformer_with_hifigan")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download espnet/fastspeech2_conformer_with_hifigan

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download espnet/fastspeech2_conformer_with_hifigan config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('espnet/fastspeech2_conformer_with_hifigan')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/espnet/fastspeech2_conformer_with_hifigan

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/espnet/fastspeech2_conformer_with_hifigan

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('espnet/fastspeech2_conformer_with_hifigan')
tokenizer = AutoTokenizer.from_pretrained('espnet/fastspeech2_conformer_with_hifigan')

完整文档

来源: HuggingFace

---
license: apache-2.0
language:

  • en

library_name: transformers
---

FastSpeech2ConformerWithHifiGan

<!-- Provide a quick summary of what the model is/does. -->

This model combines FastSpeech2Conformer and FastSpeech2ConformerHifiGan into one model for a simpler and more convenient usage.

FastSpeech2Conformer is a non-autoregressive text-to-speech (TTS) model that combines the strengths of FastSpeech2 and the conformer architecture to generate high-quality speech from text quickly and efficiently, and the HiFi-GAN vocoder is used to turn generated mel-spectrograms into speech waveforms.

🤗 Transformers Usage

You can run FastSpeech2Conformer locally with the 🤗 Transformers library.

1. First install the 🤗 Transformers library and g2p-en:

code
pip install --upgrade pip
pip install --upgrade transformers g2p-en

2. Run inference via the Transformers modelling code with the model and hifigan combined

python
from transformers import FastSpeech2ConformerTokenizer, FastSpeech2ConformerWithHifiGan
import soundfile as sf

tokenizer = FastSpeech2ConformerTokenizer.from_pretrained("espnet/fastspeech2_conformer")
inputs = tokenizer("Hello, my dog is cute.", return_tensors="pt")
input_ids = inputs["input_ids"]


model = FastSpeech2ConformerWithHifiGan.from_pretrained("espnet/fastspeech2_conformer_with_hifigan")
output_dict = model(input_ids, return_dict=True)
waveform = output_dict["waveform"]


sf.write("speech.wav", waveform.squeeze().detach().numpy(), samplerate=22050)

查看官方来源