Overview

Pyannote Segmentation MLX is a specialized voice activity detection (VAD) and speaker segmentation model optimized for Apple Silicon via the MLX framework. Unlike general-purpose audio models, this implementation focuses on high-precision boundary detection to identify when speech starts and ends, as well as distinguishing between different speakers in a stream. For developers, this means significantly lower latency and reduced memory overhead when deploying audio processing pipelines on macOS. It is an ideal component for building local transcription services, automated meeting summaries, or any application requiring robust diarization without relying on cloud APIs. By leveraging MLX, it integrates seamlessly into Python-based workflows optimized for M-series chips, offering a performant alternative to standard PyTorch deployments.

Highlights

Native MLX optimization for Apple Silicon hardware
High-precision voice activity and speaker segmentation
Low-latency local execution for privacy-focused apps
Seamless integration into Python audio pipelines
Permissive MIT license for commercial flexibility

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("aufklarer/Pyannote-Segmentation-MLX")
tokenizer = AutoTokenizer.from_pretrained("aufklarer/Pyannote-Segmentation-MLX")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download aufklarer/Pyannote-Segmentation-MLX

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download aufklarer/Pyannote-Segmentation-MLX config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('aufklarer/Pyannote-Segmentation-MLX')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/aufklarer/Pyannote-Segmentation-MLX

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/aufklarer/Pyannote-Segmentation-MLX

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('aufklarer/Pyannote-Segmentation-MLX')
tokenizer = AutoTokenizer.from_pretrained('aufklarer/Pyannote-Segmentation-MLX')

Model Download

We recommend downloading the model via the ModelScope CLI or SDK.

Guidance：Before downloading, install ModelScope with:

Guidance

pip install modelscope

CLI Download

Download the full repository

modelscope download --model aufklarer/Pyannote-Segmentation-MLX

Download a single file to a local folder (e.g. README.md into ./dir)

modelscope download --model aufklarer/Pyannote-Segmentation-MLX README.md --local_dir ./dir

See the docs for more CLI options

SDK Download

# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('aufklarer/Pyannote-Segmentation-MLX')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://www.modelscope.cn/aufklarer/Pyannote-Segmentation-MLX.git

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/aufklarer/Pyannote-Segmentation-MLX.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook Quickstart

Install the ModelScope library

pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

Load the model and run inference

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'aufklarer/Pyannote-Segmentation-MLX')

Full Documentation

来源: HuggingFace

---
license: mit
tags:
- mlx
- voice-activity-detection
- speaker-segmentation
- speaker-diarization
- pyannote
- apple-silicon
base_model: pyannote/segmentation-3.0
library_name: mlx
pipeline_tag: voice-activity-detection
---

Pyannote Segmentation 3.0 — MLX

MLX-compatible weights for pyannote/segmentation-3.0 (PyanNet), converted from the official PyTorch Lightning checkpoint with pre-computed SincNet filters.

Model

PyanNet is a speaker segmentation model (~1.5M params) that processes 10-second audio windows and outputs 7-class powerset probabilities for up to 3 simultaneous speakers. Used for both voice activity detection (binary) and speaker diarization (per-speaker).

Architecture: SincNet → BiLSTM(4 layers) → Linear(2 layers) → 7-class softmax

Output classes: non-speech, spk1, spk2, spk3, spk1+2, spk1+3, spk2+3

Usage (Swift / MLX)

swift

import SpeechVAD
// Voice Activity Detection
let vad = try await PyannoteVADModel.fromPretrained()
let segments = vad.detectSpeech(audio: samples, sampleRate: 16000)
for seg in segments {
    print("Speech: \(seg.startTime)s - \(seg.endTime)s")
}
// Speaker Diarization (with WeSpeaker embeddings)
let pipeline = try await DiarizationPipeline.fromPretrained()
let result = pipeline.diarize(audio: samples, sampleRate: 16000)
for seg in result.segments {
    print("Speaker \(seg.speakerId): \(seg.startTime)s - \(seg.endTime)s")
}

Part of speech-swift.

Conversion

bash

python3 scripts/convert_pyannote.py --token YOUR_HF_TOKEN --upload

Converts the gated pyannote/segmentation-3.0 checkpoint using a custom unpickler (no pyannote.audio dependency required). Key transformations:

SincNet: pre-compute 80 sinc bandpass filters (40 cos + 40 sin) from 40 learned (low_hz, band_hz) parameter pairs

Conv1d: transpose weights [O, I, K] → [O, K, I] for MLX channels-last

BiLSTM: split into forward/backward stacks, sum bias_ih + bias_hh

Linear/classifier: kept as-is

Weight Mapping

| PyTorch Key | MLX Key | Shape |
|-------------|---------|-------|
| sincnet.conv1d.0.filterbank.* (computed) | sincnet.conv.0.weight | [80, 251, 1] |
| sincnet.conv1d.{1,2}.weight | sincnet.conv.{1,2}.weight | [O, K, I] |
| sincnet.norm1d.{0-2}.* | sincnet.norm.{0-2}.* | varies |
| lstm.weight_ih_l{i} | lstm_fwd.layers.{i}.Wx | [512, I] |
| lstm.weight_hh_l{i} | lstm_fwd.layers.{i}.Wh | [512, 128] |
| lstm.bias_ih_l{i} + bias_hh_l{i} | lstm_fwd.layers.{i}.bias | [512] |
| lstm.*_reverse | lstm_bwd.layers.{i}.* | same |
| linear.{0,1}.* | linear.{0,1}.* | varies |
| classifier.* | classifier.* | [7, 128] |

License

The original pyannote segmentation model is released under the MIT License.

---

Guide: soniqo.audio/guides/diarize

Docs: soniqo.audio

GitHub: soniqo/speech-swift