whisper audio to text

提供商AventIQ-AI
分类audio-generation
许可证Apache-2.0
下载量5
星标0

简介

Whisper 是一款由 OpenAI 开源、目前被公认为工业级标杆的语音转文字模型。它通过在大规模多语言数据集上进行训练，实现了极强的鲁棒性，即使在背景噪音较大或口音较重的情况下，也能提供极高准确率的转录和翻译。对于开发者而言，它不仅支持多种语言的实时转写，还可通过本地部署确保数据隐私。相比于传统的 API 语音服务，Whisper 的上手门槛较低，且由于其开源生态丰富，很容易集成到自动化工作流或会议记录工具中。

核心亮点

  • 多语言精准转录，支持高效语音翻译
  • 抗噪能力极强，适配复杂真实场景
  • 开源协议灵活，支持本地私有化部署
  • 集成简单，是构建语音应用的理想底座

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("AventIQ-AI/whisper-audio-to-text")
tokenizer = AutoTokenizer.from_pretrained("AventIQ-AI/whisper-audio-to-text")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download AventIQ-AI/whisper-audio-to-text

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download AventIQ-AI/whisper-audio-to-text config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('AventIQ-AI/whisper-audio-to-text')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/AventIQ-AI/whisper-audio-to-text

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/AventIQ-AI/whisper-audio-to-text

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('AventIQ-AI/whisper-audio-to-text')
tokenizer = AutoTokenizer.from_pretrained('AventIQ-AI/whisper-audio-to-text')

完整文档

来源: HuggingFace

OpenAI Whisper-Base Fine-Tuned Model for Speech-to-Text

This repository hosts a fine-tuned version of the OpenAI Whisper-Base model optimized for speech-to-text tasks using the Mozilla Common Voice 13.0 dataset. The model is designed to efficiently transcribe speech into text while maintaining high accuracy.

Model Details

  • Model Architecture: OpenAI Whisper-Base
  • Task: Audio-to-Text
  • Fine-tuning Framework: Hugging Face Transformers

🚀 Usage

Installation

bash
pip install transformers torch

Loading the Model

python
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"


model_name = "AventIQ-AI/whisper-audio-to-text"
model = WhisperForConditionalGeneration.from_pretrained(model_name).to(device)
processor = WhisperProcessor.from_pretrained(model_name)

Speech-to-Text Inference

python
import torchaudio

Load and process audio file

def load_audio(file_path, target_sampling_rate=16000):
    # Load audio file
    waveform, sample_rate = torchaudio.load(file_path)

# Convert to mono if stereo
    if waveform.shape[0] > 1:
        waveform = waveform.mean(dim=0, keepdim=True)


# Resample if needed
    if sample_rate != target_sampling_rate:
        waveform = torchaudio.transforms.Resample(orig_freq=sample_rate, new_freq=target_sampling_rate)(waveform)


return waveform.squeeze(0).numpy()


input_audio_path = "/kaggle/input/test-data-2/Friday 4h04m pm.m4a"  # Change this to your audio file
audio_array = load_audio(input_audio_path)


input_features = processor(audio_array, sampling_rate=16000, return_tensors="pt").input_features
input_features = input_features.to(device)


forced_decoder_ids = processor.get_decoder_prompt_ids(language="en", task="transcribe")


with torch.no_grad():
    predicted_ids = model.generate(input_features, forced_decoder_ids=forced_decoder_ids)


Decode output

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]

print(f"Transcribed Text: {transcription}")

📊 Evaluation Results

After fine-tuning the Whisper-Base model for speech-to-text, we evaluated the model's performance on the validation set from the Common Voice 11.0 dataset. The following results were obtained:

| Metric | Score | Meaning |
|------------|--------|------------------------------------------------|
| WER | 9.2% | Word Error Rate: Measures transcription accuracy |
| CER | 5.5% | Character Error Rate: Measures character-level accuracy |

Fine-Tuning Details

Dataset

The Mozilla Common Voice 11.0 dataset, containing diverse multilingual speech samples, was used for fine-tuning the model.

Training

  • Number of epochs: 6
  • Batch size: 16
  • Evaluation strategy: epochs
  • Learning Rate: 5e-6

📂 Repository Structure

bash
.
├── model/               # Contains the quantized model files
├── tokenizer_config/    # Tokenizer configuration and vocabulary files
├── model.safetensors/   # Quantized Model
├── README.md            # Model documentation

⚠️ Limitations

  • The model may struggle with highly noisy or overlapping speech.
  • Performance may vary across different accents and dialects.

🤝 Contributing

Contributions are welcome! Feel free to open an issue or submit a pull request if you have suggestions or improvements.
查看官方来源