Pyannote Segmentation MLX

提供商aufklarer
分类voice-activity-detection
许可证mit
下载量159
星标0

简介

Pyannote Segmentation MLX 是专为 Apple Silicon 芯片优化的语音活动检测（VAD）模型。它将 Pyannote 强大的语音分割能力通过 MLX 框架迁移至 Mac 端，旨在解决音频预处理中精准识别“有人说话”与“静默/噪音”的痛点。对于开发者而言，它无需依赖沉重的 PyTorch 环境即可在本地高效运行，非常适合用于播客剪辑自动化、会议记录预处理等需要快速剔除空白音频的场景。上手门槛较低，是构建本地化语音管线（如结合 Whisper 进行转写）时理想的轻量化前置组件。

核心亮点

  • 原生支持 Apple Silicon，端侧推理速度极快
  • 精准区分语音与静默，大幅提升转写效率
  • 轻量化部署，无需复杂环境即可本地运行
  • 是 Whisper 等语音转写工具的绝佳前置插件

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("aufklarer/Pyannote-Segmentation-MLX")
tokenizer = AutoTokenizer.from_pretrained("aufklarer/Pyannote-Segmentation-MLX")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download aufklarer/Pyannote-Segmentation-MLX

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download aufklarer/Pyannote-Segmentation-MLX config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('aufklarer/Pyannote-Segmentation-MLX')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/aufklarer/Pyannote-Segmentation-MLX

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/aufklarer/Pyannote-Segmentation-MLX

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('aufklarer/Pyannote-Segmentation-MLX')
tokenizer = AutoTokenizer.from_pretrained('aufklarer/Pyannote-Segmentation-MLX')

模型下载

我们推荐使用命令行或者 ModelScope SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 ModelScope：

操作指引 
pip install modelscope

命令行下载

下载完整模型库

下载完整模型库 
modelscope download --model aufklarer/Pyannote-Segmentation-MLX

下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例）

下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例） 
modelscope download --model aufklarer/Pyannote-Segmentation-MLX README.md --local_dir ./dir

更多更丰富的命令行下载选项，可参见具体文档

SDK 下载

SDK 下载 
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('aufklarer/Pyannote-Segmentation-MLX')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://www.modelscope.cn/aufklarer/Pyannote-Segmentation-MLX.git

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/aufklarer/Pyannote-Segmentation-MLX.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook 快速开发

下载并安装 ModelScope library

下载并安装 ModelScope library 
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

模型加载和推理

模型加载和推理 
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'aufklarer/Pyannote-Segmentation-MLX')

完整文档

来源: HuggingFace

---
license: mit
tags:
- mlx
- voice-activity-detection
- speaker-segmentation
- speaker-diarization
- pyannote
- apple-silicon
base_model: pyannote/segmentation-3.0
library_name: mlx
pipeline_tag: voice-activity-detection
---

Pyannote Segmentation 3.0 — MLX

MLX-compatible weights for pyannote/segmentation-3.0 (PyanNet), converted from the official PyTorch Lightning checkpoint with pre-computed SincNet filters.

Model

PyanNet is a speaker segmentation model (~1.5M params) that processes 10-second audio windows and outputs 7-class powerset probabilities for up to 3 simultaneous speakers. Used for both voice activity detection (binary) and speaker diarization (per-speaker).

Architecture: SincNet → BiLSTM(4 layers) → Linear(2 layers) → 7-class softmax

Output classes: non-speech, spk1, spk2, spk3, spk1+2, spk1+3, spk2+3

Usage (Swift / MLX)

swift
import SpeechVAD

// Voice Activity Detection
let vad = try await PyannoteVADModel.fromPretrained()
let segments = vad.detectSpeech(audio: samples, sampleRate: 16000)
for seg in segments {
    print("Speech: \(seg.startTime)s - \(seg.endTime)s")
}


// Speaker Diarization (with WeSpeaker embeddings)
let pipeline = try await DiarizationPipeline.fromPretrained()
let result = pipeline.diarize(audio: samples, sampleRate: 16000)
for seg in result.segments {
    print("Speaker \(seg.speakerId): \(seg.startTime)s - \(seg.endTime)s")
}

Part of speech-swift.

Conversion

bash
python3 scripts/convert_pyannote.py --token YOUR_HF_TOKEN --upload

Converts the gated pyannote/segmentation-3.0 checkpoint using a custom unpickler (no pyannote.audio dependency required). Key transformations:

  • SincNet: pre-compute 80 sinc bandpass filters (40 cos + 40 sin) from 40 learned (low_hz, band_hz) parameter pairs
  • Conv1d: transpose weights [O, I, K] → [O, K, I] for MLX channels-last
  • BiLSTM: split into forward/backward stacks, sum bias_ih + bias_hh
  • Linear/classifier: kept as-is

Weight Mapping

| PyTorch Key | MLX Key | Shape |
|-------------|---------|-------|
| sincnet.conv1d.0.filterbank.* (computed) | sincnet.conv.0.weight | [80, 251, 1] |
| sincnet.conv1d.{1,2}.weight | sincnet.conv.{1,2}.weight | [O, K, I] |
| sincnet.norm1d.{0-2}.* | sincnet.norm.{0-2}.* | varies |
| lstm.weight_ih_l{i} | lstm_fwd.layers.{i}.Wx | [512, I] |
| lstm.weight_hh_l{i} | lstm_fwd.layers.{i}.Wh | [512, 128] |
| lstm.bias_ih_l{i} + bias_hh_l{i} | lstm_fwd.layers.{i}.bias | [512] |
| lstm.*_reverse | lstm_bwd.layers.{i}.* | same |
| linear.{0,1}.* | linear.{0,1}.* | varies |
| classifier.* | classifier.* | [7, 128] |

License

The original pyannote segmentation model is released under the MIT License.

---

---

查看官方来源