speaker diarization coreml

提供商FluidInference
分类voice-activity-detection
许可证cc-by-4.0
下载量13.3K
星标0

简介

这是一个基于 CoreML 框架实现的说话人日志（Speaker Diarization）模型，由 FluidInference 提供。它主要解决的是“谁在什么时候说话”的问题，能将音频流中的不同发言人进行自动区分和标记。由于采用了 CoreML 优化，该模型能够高效运行在 Apple 芯片的设备上，充分利用 NPU 加速，无需依赖云端 API 即可实现低延迟的本地化处理。对于需要开发 iOS/macOS 端语音转写、会议记录或播客剪辑工具的开发者来说，这是一个极佳的轻量化端侧方案。

核心亮点

  • 原生支持 CoreML，在 Apple 设备上运行极快
  • 实现本地化说话人区分，无需上传数据，隐私性高
  • 适用于会议记录、采访转录等多角色语音场景
  • 轻量化部署，降低了端侧语音分析的上手难度

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("FluidInference/speaker-diarization-coreml")
tokenizer = AutoTokenizer.from_pretrained("FluidInference/speaker-diarization-coreml")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download FluidInference/speaker-diarization-coreml

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download FluidInference/speaker-diarization-coreml config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('FluidInference/speaker-diarization-coreml')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/FluidInference/speaker-diarization-coreml

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/FluidInference/speaker-diarization-coreml

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('FluidInference/speaker-diarization-coreml')
tokenizer = AutoTokenizer.from_pretrained('FluidInference/speaker-diarization-coreml')

完整文档

来源: HuggingFace

---
license: cc-by-4.0
tags:

  • speech

  • audio

  • voice

  • speaker-diarization

  • speaker-change-detection

  • coreml

  • speaker-segmentation

base_model:
  • pyannote/speaker-diarization-community-1

base_model_relation: finetune
pipeline_tag: voice-activity-detection
---

<span style="color:#5DAF8D">🧃 Speaker Diarization CoreML </span>

![Discord](https://discord.gg/WNsvaCtmDe) ![GitHub Repo stars](https://github.com/FluidInference/FluidAudio)

Speaker diarization based on pyannote models optimized for Apple Neural Engine.

Models are trained on acoustic signatures so it supports any lanugage.

Usage

See the SDK for more details https://github.com/FluidInference/FluidAudio

Please note that the SDK itself is Apache 2.0, but the parent model from Pyannote is cc-by-4.0

Technical Specifications

  • Input: 16kHz mono audio
  • Output: Speaker segments with timestamps and IDs
  • Framework: CoreML (converted from PyTorch)
  • Optimization: Apple Neural Engine (ANE) optimized operations
  • Precision: FP32 on CPU/GPU, FP16 on ANE

Performance

See the origianl model for detailed DER benchmark, for the purpose of our conversion, we tried to match the original model as much as possible:

The models on CoreML exhibit a ~10x Speedup on CPU and ~20x speed up on GPU.

!plots/pipeline_timing.png

Due to different precisions, there are minor differences in the values generated but the differences are mostly negilible, though it does account for some errors that needs to be adjusted during clustering:

!plots/metrics_timeseries.png

We see this when running the end to end pipeline with the Pytorch model versus the Core ML model (patched the Pyannote pipeline to run the Core ML model instead). The DER and JER is ~1% compared to the Pytorch model as we're dropping the precision to fp32
!plots/pipeline_overview.png

Citations (from original model)

1. Speaker segmentation model

bibtex
@inproceedings{Plaquet23,
  author={Alexis Plaquet and Hervé Bredin},
  title={{Powerset multi-class cross entropy loss for neural speaker diarization}},
  year=2023,
  booktitle={Proc. INTERSPEECH 2023},
}

2. Speaker embedding model

bibtex
@inproceedings{Wang2023,
  title={Wespeaker: A research and production oriented speaker embedding learning toolkit},
  author={Wang, Hongji and Liang, Chengdong and Wang, Shuai and Chen, Zhengyang and Zhang, Binbin and Xiang, Xu and Deng, Yanlei and Qian, Yanmin},
  booktitle={ICASSP 2023, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={1--5},
  year={2023},
  organization={IEEE}
}

3. Speaker clustering

```bibtex
@article{Landini2022,
author={Landini, Federico and Profant, J{\'a}n and Diez, Mireia and Burget, Luk{\'a}{\v{s}}},
title={{Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks}},
year={2022},
journal={Computer Speech \& Language},
}

查看官方来源