smart turn v2
简介
核心亮点
- 精准识别语音起止，优化实时对话的轮替时机
- 低延迟响应，有效解决语音助手被误打断问题
- 轻量化部署，完美集成于语音交互流水线
- BSD 许可协议，对商业应用非常友好
使用方法
# 安装 Hugging Face transformers
pip install transformers torch# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("pipecat-ai/smart-turn-v2")
tokenizer = AutoTokenizer.from_pretrained("pipecat-ai/smart-turn-v2")Hugging Face 下载
我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。
操作指引：在下载前，请先通过如下命令安装 huggingface_hub：
pip install -U huggingface_hub命令行下载
下载完整模型库
huggingface-cli download pipecat-ai/smart-turn-v2下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）
huggingface-cli download pipecat-ai/smart-turn-v2 config.json --local-dir ./dirSDK 下载
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('pipecat-ai/smart-turn-v2')Git 下载
请确保 lfs 已经被正确安装
git lfs install
git clone https://huggingface.co/pipecat-ai/smart-turn-v2如果您希望跳过 lfs 大文件下载，可以使用如下命令
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/pipecat-ai/smart-turn-v2模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。
PyTorch / Transformers 使用
安装 Transformers
pip install -U transformers torch模型加载和推理
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('pipecat-ai/smart-turn-v2')
tokenizer = AutoTokenizer.from_pretrained('pipecat-ai/smart-turn-v2')模型下载
我们推荐使用命令行或者 ModelScope SDK 来进行模型的下载。
操作指引：在下载前，请先通过如下命令安装 ModelScope：
pip install modelscope命令行下载
下载完整模型库
modelscope download --model pipecat-ai/smart-turn-v2下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例）
modelscope download --model pipecat-ai/smart-turn-v2 README.md --local_dir ./dirSDK 下载
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('pipecat-ai/smart-turn-v2')Git 下载
请确保 lfs 已经被正确安装
git lfs install
git clone https://www.modelscope.cn/pipecat-ai/smart-turn-v2.git如果您希望跳过 lfs 大文件下载，可以使用如下命令
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/pipecat-ai/smart-turn-v2.gitModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。
Notebook 快速开发
下载并安装 ModelScope library
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html模型加载和推理
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks
p = pipeline('text-generation', 'pipecat-ai/smart-turn-v2')完整文档
---
pipeline_tag: voice-activity-detection
license: bsd-2-clause
tags:
- speech-processing
- semantic-vad
- multilingual
datasets:
- pipecat-ai/chirp3_1
- pipecat-ai/orpheus_midfiller_1
- pipecat-ai/orpheus_grammar_1
- pipecat-ai/orpheus_endfiller_1
- pipecat-ai/human_convcollector_1
- pipecat-ai/rime_2
- pipecat-ai/human_5_all
languages:
- en
- fr
- de
- es
- pt
- zh
- ja
- hi
- it
- ko
- nl
- pl
- ru
- tr
---
Smart Turn v2
Smart Turn v2 is an open‑source semantic Voice Activity Detection (VAD) model that tells you _whether a speaker has finished their turn_ by analysing the raw waveform, not the transcript.
Compared with v1 it is:
- Multilingual – 14 languages (EN, FR, DE, ES, PT, ZH, JA, HI, IT, KO, NL, PL, RU, TR).
- 6 × smaller – ≈ 360 MB vs. 2.3 GB.
- 3 × faster – ≈ 12 ms to analyse 8 s of audio on an NVIDIA L40S.
Links
- GitHub repo with training and inference code
Intended use & task
| Use‑case | Why this model helps |
|---------------------------------------------|-------------------------------------------------------------------------|
| Voice agents / chatbots | Wait to reply until the user has actually finished speaking. |
| Real‑time transcription + TTS | Avoid “double‑talk” by triggering TTS only when the user turn ends. |
| Call‑centre assist & analytics | Accurate segmentation for diarisation and sentiment pipelines. |
| Any project needing semantic VAD | Detects incomplete thoughts, filler words (“um …”, “えーと …”) and intonation cues ignored by classic energy‑based VAD. |
The model outputs a single probability; values ≥ 0.5 indicate the speaker has completed their utterance.
Model architecture
- Backbone :
wav2vec2encoder
- Head : shallow linear classifier
- Params : 94.8 M (float32)
- Checkpoint: 360 MB Safetensors (compressed)
wav2vec2 + linear configuration out‑performed LSTM and deeper transformer variants during ablation studies.
Training data
| Source | Type | Languages |
|-------------------------|-------------------------------|-----------|
| human_5_all | Human‑recorded | EN |
| human_convcollector_1 | Human‑recorded | EN |
| rime_2 | Synthetic (Rime) | EN |
| orpheus_midfiller_1 | Synthetic (Orpheus) | EN |
| orpheus_grammar_1 | Synthetic (Orpheus) | EN |
| orpheus_endfiller_1 | Synthetic (Orpheus) | EN |
| chirp3_1 | Synthetic (Google Chirp3 TTS) | 14 langs |
- Sentences were cleaned with Gemini 2.5 Flash to remove ungrammatical, controversial or written‑only text.
- Filler‑word lists per language (e.g., “um”, “えーと”) built with Claude & GPT‑o3 and injected near sentence ends to teach the model about interrupted speech.
All audio/text pairs are released on the pipecat‑ai/datasets hub.
Evaluation & performance
Accuracy on unseen synthetic test set (50 % complete / 50 % incomplete)
| Lang | Acc % | Lang | Acc % | |------|-------|------|-------| | EN | 94.3 | IT | 94.4 | | FR | 95.5 | KO | 95.5 | | ES | 92.1 | PT | 95.5 | | DE | 95.8 | TR | 96.8 | | NL | 96.7 | PL | 94.6 | | RU | 93.0 | HI | 91.2 | | ZH | 87.2 | – | – |*Human English benchmark (human_5_all) : 99 % accuracy.*
Inference latency for 8 s audio
| Device | Time |
|-------------------------------|------|
| NVIDIA L40S | 12 ms |
| NVIDIA A100 | 19 ms |
| NVIDIA T4 (AWS g4dn.xlarge) | 75 ms |
| 16‑core x86\_64 CPU (Modal) | 410 ms |
How to use
Please see the blog post and GitHub repo for more information on using the model, either standalone or with Pipecat.
