smart turn v2

Providerpipecat-ai
Categoryvoice-activity-detection
Licensebsd-2-clause
Downloads254
Stars3

Overview

Smart Turn v2 is a specialized Voice Activity Detection (VAD) model designed to optimize the interaction flow in real-time conversational AI. For developers building voice agents, the primary challenge is distinguishing between natural pauses and actual turn-taking cues to avoid awkward interruptions or excessive latency. This model focuses on high-precision endpointing, allowing applications to trigger responses more accurately based on speaker intent rather than simple silence thresholds. It integrates seamlessly into pipelines requiring low-latency audio processing and is released under the permissive BSD-2-Clause license, making it an ideal lightweight component for scalable, production-ready voice interfaces.

Highlights

  • Optimizes turn-taking for low-latency conversational AI
  • Reduces false triggers during natural speech pauses
  • Permissive BSD-2-Clause license for commercial deployment
  • Lightweight integration for real-time audio pipelines

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("pipecat-ai/smart-turn-v2")
tokenizer = AutoTokenizer.from_pretrained("pipecat-ai/smart-turn-v2")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance 
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository 
huggingface-cli download pipecat-ai/smart-turn-v2

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir) 
huggingface-cli download pipecat-ai/smart-turn-v2 config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('pipecat-ai/smart-turn-v2')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://huggingface.co/pipecat-ai/smart-turn-v2

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/pipecat-ai/smart-turn-v2

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers 
pip install -U transformers torch

Load the model and run inference

Load the model and run inference 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('pipecat-ai/smart-turn-v2')
tokenizer = AutoTokenizer.from_pretrained('pipecat-ai/smart-turn-v2')

Model Download

We recommend downloading the model via the ModelScope CLI or SDK.

Guidance：Before downloading, install ModelScope with:

Guidance 
pip install modelscope

CLI Download

Download the full repository

Download the full repository 
modelscope download --model pipecat-ai/smart-turn-v2

Download a single file to a local folder (e.g. README.md into ./dir)

Download a single file to a local folder (e.g. README.md into ./dir) 
modelscope download --model pipecat-ai/smart-turn-v2 README.md --local_dir ./dir

See the docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('pipecat-ai/smart-turn-v2')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://www.modelscope.cn/pipecat-ai/smart-turn-v2.git

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/pipecat-ai/smart-turn-v2.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook Quickstart

Install the ModelScope library

Install the ModelScope library 
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

Load the model and run inference

Load the model and run inference 
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'pipecat-ai/smart-turn-v2')

Full Documentation

来源: HuggingFace

---
pipeline_tag: voice-activity-detection
license: bsd-2-clause
tags:
- speech-processing
- semantic-vad
- multilingual
datasets:
- pipecat-ai/chirp3_1
- pipecat-ai/orpheus_midfiller_1
- pipecat-ai/orpheus_grammar_1
- pipecat-ai/orpheus_endfiller_1
- pipecat-ai/human_convcollector_1
- pipecat-ai/rime_2
- pipecat-ai/human_5_all
languages:
- en
- fr
- de
- es
- pt
- zh
- ja
- hi
- it
- ko
- nl
- pl
- ru
- tr
---

Smart Turn v2

Smart Turn v2 is an open‑source semantic Voice Activity Detection (VAD) model that tells you _whether a speaker has finished their turn_ by analysing the raw waveform, not the transcript.
Compared with v1 it is:

  • Multilingual – 14 languages (EN, FR, DE, ES, PT, ZH, JA, HI, IT, KO, NL, PL, RU, TR).
  • 6 × smaller – ≈ 360 MB vs. 2.3 GB.
  • 3 × faster – ≈ 12 ms to analyse 8 s of audio on an NVIDIA L40S.

Links

Intended use & task

| Use‑case | Why this model helps |
|---------------------------------------------|-------------------------------------------------------------------------|
| Voice agents / chatbots | Wait to reply until the user has actually finished speaking. |
| Real‑time transcription + TTS | Avoid “double‑talk” by triggering TTS only when the user turn ends. |
| Call‑centre assist & analytics | Accurate segmentation for diarisation and sentiment pipelines. |
| Any project needing semantic VAD | Detects incomplete thoughts, filler words (“um …”, “えーと …”) and intonation cues ignored by classic energy‑based VAD. |

The model outputs a single probability; values ≥ 0.5 indicate the speaker has completed their utterance.

Model architecture

  • Backbone : wav2vec2 encoder
  • Head     : shallow linear classifier
  • Params   : 94.8 M (float32)
  • Checkpoint: 360 MB Safetensors (compressed)
The wav2vec2 + linear configuration out‑performed LSTM and deeper transformer variants during ablation studies.

Training data

| Source | Type | Languages |
|-------------------------|-------------------------------|-----------|
| human_5_all | Human‑recorded | EN |
| human_convcollector_1 | Human‑recorded | EN |
| rime_2 | Synthetic (Rime) | EN |
| orpheus_midfiller_1 | Synthetic (Orpheus) | EN |
| orpheus_grammar_1 | Synthetic (Orpheus) | EN |
| orpheus_endfiller_1 | Synthetic (Orpheus) | EN |
| chirp3_1 | Synthetic (Google Chirp3 TTS) | 14 langs |

  • Sentences were cleaned with Gemini 2.5 Flash to remove ungrammatical, controversial or written‑only text.
  • Filler‑word lists per language (e.g., “um”, “えーと”) built with Claude & GPT‑o3 and injected near sentence ends to teach the model about interrupted speech.

All audio/text pairs are released on the pipecat‑ai/datasets hub.

Evaluation & performance

Accuracy on unseen synthetic test set (50 % complete / 50 % incomplete)

| Lang | Acc % | Lang | Acc % | |------|-------|------|-------| | EN | 94.3 | IT | 94.4 | | FR | 95.5 | KO | 95.5 | | ES | 92.1 | PT | 95.5 | | DE | 95.8 | TR | 96.8 | | NL | 96.7 | PL | 94.6 | | RU | 93.0 | HI | 91.2 | | ZH | 87.2 | – | – |

*Human English benchmark (human_5_all) : 99 % accuracy.*

Inference latency for 8 s audio

| Device | Time |
|-------------------------------|------|
| NVIDIA L40S | 12 ms |
| NVIDIA A100 | 19 ms |
| NVIDIA T4 (AWS g4dn.xlarge) | 75 ms |
| 16‑core x86\_64 CPU (Modal) | 410 ms |

oai_citation:7‡Daily

How to use

Please see the blog post and GitHub repo for more information on using the model, either standalone or with Pipecat.

View official source
Join our Telegram