Name: speaker diarization coreml
Author: FluidInference

Overview

Speaker Diarization CoreML is a specialized model optimized for on-device audio processing, specifically designed to identify 'who spoke when' within an audio stream. Unlike general-purpose ASR, this model focuses on speaker segmentation and identity clustering, making it ideal for developers building local transcription services, meeting summaries, or voice-controlled interfaces where privacy and low latency are critical. By leveraging CoreML, it offloads computation to the Apple Neural Engine (ANE), significantly reducing CPU overhead and battery drain compared to standard PyTorch or TensorFlow implementations. It integrates seamlessly into iOS and macOS workflows, providing a performant alternative to cloud-based diarization APIs for real-time or offline applications.

Highlights

Optimized for Apple Neural Engine and on-device execution
Efficient speaker segmentation and identity clustering
Low-latency processing for privacy-focused local applications
Reduced battery consumption via CoreML hardware acceleration
Open CC-BY-4.0 license for flexible commercial integration

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("FluidInference/speaker-diarization-coreml")
tokenizer = AutoTokenizer.from_pretrained("FluidInference/speaker-diarization-coreml")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download FluidInference/speaker-diarization-coreml

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download FluidInference/speaker-diarization-coreml config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('FluidInference/speaker-diarization-coreml')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/FluidInference/speaker-diarization-coreml

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/FluidInference/speaker-diarization-coreml

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('FluidInference/speaker-diarization-coreml')
tokenizer = AutoTokenizer.from_pretrained('FluidInference/speaker-diarization-coreml')

Full Documentation

来源: HuggingFace

---
license: cc-by-4.0
tags:

speech

audio

voice

speaker-diarization

speaker-change-detection

coreml

speaker-segmentation

base_model:

pyannote/speaker-diarization-community-1

base_model_relation: finetune
pipeline_tag: voice-activity-detection
---

<span style="color:#5DAF8D">🧃 Speaker Diarization CoreML </span>

![Discord](https://discord.gg/WNsvaCtmDe) ![GitHub Repo stars](https://github.com/FluidInference/FluidAudio)

Speaker diarization based on pyannote models optimized for Apple Neural Engine.

Models are trained on acoustic signatures so it supports any lanugage.

Usage

See the SDK for more details https://github.com/FluidInference/FluidAudio

Please note that the SDK itself is Apache 2.0, but the parent model from Pyannote is cc-by-4.0

Technical Specifications

Input: 16kHz mono audio

Output: Speaker segments with timestamps and IDs

Framework: CoreML (converted from PyTorch)

Optimization: Apple Neural Engine (ANE) optimized operations

Precision: FP32 on CPU/GPU, FP16 on ANE

Performance

See the origianl model for detailed DER benchmark, for the purpose of our conversion, we tried to match the original model as much as possible:

The models on CoreML exhibit a ~10x Speedup on CPU and ~20x speed up on GPU.

!plots/pipeline_timing.png

Due to different precisions, there are minor differences in the values generated but the differences are mostly negilible, though it does account for some errors that needs to be adjusted during clustering:

!plots/metrics_timeseries.png

We see this when running the end to end pipeline with the Pytorch model versus the Core ML model (patched the Pyannote pipeline to run the Core ML model instead). The DER and JER is ~1% compared to the Pytorch model as we're dropping the precision to fp32
!plots/pipeline_overview.png

Citations (from original model)

1. Speaker segmentation model

bibtex

@inproceedings{Plaquet23,
  author={Alexis Plaquet and Hervé Bredin},
  title={{Powerset multi-class cross entropy loss for neural speaker diarization}},
  year=2023,
  booktitle={Proc. INTERSPEECH 2023},
}

2. Speaker embedding model

bibtex

@inproceedings{Wang2023,
  title={Wespeaker: A research and production oriented speaker embedding learning toolkit},
  author={Wang, Hongji and Liang, Chengdong and Wang, Shuai and Chen, Zhengyang and Zhang, Binbin and Xiang, Xu and Deng, Yanlei and Qian, Yanmin},
  booktitle={ICASSP 2023, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={1--5},
  year={2023},
  organization={IEEE}
}

3. Speaker clustering

```bibtex
@article{Landini2022,
author={Landini, Federico and Profant, J{\'a}n and Diez, Mireia and Burget, Luk{\'a}{\v{s}}},
title={{Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: theory, implementation and analysis on standard tasks}},
year={2022},
journal={Computer Speech \& Language},
}