wav2vec2 indonesian javanese sundanese

提供商indonesian-nlp
分类automatic-speech-recognition
许可证apache-2.0
下载量2.2M
星标0

简介

这是一个基于 wav2vec2 架构的专项语音识别模型，专注于印尼语及其主要方言（爪哇语和巽他语）的转写。对于需要处理东南亚多语种语音数据的开发者来说，它解决了通用模型在印尼地方语言上识别率低的问题。该模型上手门槛较低，可通过 Hugging Face 等主流框架快速部署，适用于构建当地语言的语音助手、会议记录工具或多语言字幕翻译管线，是处理印尼语系语音任务的实用基准选择。

核心亮点

  • 精准覆盖印尼语、爪哇语和巽他语三语
  • 基于 wav2vec2 架构，适配主流 AI 框架
  • 适用于东南亚多语种语音转文字场景
  • Apache-2.0 开源协议，商业部署灵活

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("indonesian-nlp/wav2vec2-indonesian-javanese-sundanese")
tokenizer = AutoTokenizer.from_pretrained("indonesian-nlp/wav2vec2-indonesian-javanese-sundanese")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download indonesian-nlp/wav2vec2-indonesian-javanese-sundanese

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download indonesian-nlp/wav2vec2-indonesian-javanese-sundanese config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('indonesian-nlp/wav2vec2-indonesian-javanese-sundanese')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/indonesian-nlp/wav2vec2-indonesian-javanese-sundanese

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/indonesian-nlp/wav2vec2-indonesian-javanese-sundanese

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('indonesian-nlp/wav2vec2-indonesian-javanese-sundanese')
tokenizer = AutoTokenizer.from_pretrained('indonesian-nlp/wav2vec2-indonesian-javanese-sundanese')

完整文档

来源: HuggingFace

---
language:

  • id

  • jv

  • sun

datasets:
  • mozilla-foundation/common_voice_7_0

  • openslr

  • magic_data

  • titml

metrics:
  • wer

tags:
  • audio

  • automatic-speech-recognition

  • hf-asr-leaderboard

  • id

  • jv

  • robust-speech-event

  • speech

  • su

license: apache-2.0
model-index:
  • name: Wav2Vec2 Indonesian Javanese and Sundanese by Indonesian NLP

results:
- task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 6.1
type: common_voice
args: id
metrics:
- name: Test WER
type: wer
value: 4.056
- name: Test CER
type: cer
value: 1.472
- task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 7
type: mozilla-foundation/common_voice_7_0
args: id
metrics:
- name: Test WER
type: wer
value: 4.492
- name: Test CER
type: cer
value: 1.577
- task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: id
metrics:
- name: Test WER
type: wer
value: 48.94
- task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Test Data
type: speech-recognition-community-v2/eval_data
args: id
metrics:
- name: Test WER
type: wer
value: 68.95
---

Multilingual Speech Recognition for Indonesian Languages

This is the model built for the project
Multilingual Speech Recognition for Indonesian Languages.
It is a fine-tuned facebook/wav2vec2-large-xlsr-53
model on the Indonesian Common Voice dataset,
High-quality TTS data for Javanese - SLR41, and
High-quality TTS data for Sundanese - SLR44 datasets.

We also provide a live demo to test the model.

When using this model, make sure that your speech input is sampled at 16kHz.

Usage

The model can be used directly (without a language model) as follows:
python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

test_dataset = load_dataset("common_voice", "id", split="test[:2%]")


processor = Wav2Vec2Processor.from_pretrained("indonesian-nlp/wav2vec2-indonesian-javanese-sundanese")
model = Wav2Vec2ForCTC.from_pretrained("indonesian-nlp/wav2vec2-indonesian-javanese-sundanese")


resampler = torchaudio.transforms.Resample(48_000, 16_000)


Preprocessing the datasets.


We need to read the aduio files as arrays

def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset[:2]["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)


with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits


predicted_ids = torch.argmax(logits, dim=-1)


print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset[:2]["sentence"])

Evaluation

The model can be evaluated as follows on the Indonesian test data of Common Voice.

python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

test_dataset = load_dataset("common_voice", "id", split="test")
wer = load_metric("wer")


processor = Wav2Vec2Processor.from_pretrained("indonesian-nlp/wav2vec2-indonesian-javanese-sundanese")
model = Wav2Vec2ForCTC.from_pretrained("indonesian-nlp/wav2vec2-indonesian-javanese-sundanese") 
model.to("cuda")


chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\‘\'\”\�]'


resampler = torchaudio.transforms.Resample(48_000, 16_000)


Preprocessing the datasets.


We need to read the audio files as arrays

def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)


Preprocessing the datasets.


We need to read the audio files as arrays

def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits


pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch


result = test_dataset.map(evaluate, batched=True, batch_size=8)


print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))

Test Result: 11.57 %

Training

The Common Voice train, validation, and ... datasets were used for training as well as ... and ... # TODO

The script used for training can be found here
(will be available soon)

查看官方来源