Voice AI Latency: Why Faster Isn't Always Better
My pipeline was hitting a p95 of about 880ms from the end of a turn to the first audio byte. On paper, that's great. In practice, it was a disaster. The AI was constantly interrupting users mid-sentence because it was too fast to respond to a brief thinking pause.
The Latency-Endpointing Trap
The core issue is that latency and endpointing are linked. When your pipeline is incredibly fast, any mistake the Voice Activity Detection (VAD) makes in deciding the turn has ended is amplified. If the AI responds in under a second, it cuts the user off. If it were slightly slower, the user would have resumed speaking, the "barge-in" mechanism would have triggered, and the conversation would have felt natural.
I was treating endpointing as a minor config detail, but it's actually the primary driver of conversational feel.
Practical Tuning for Natural Conversation
To fix the interruptions, I moved away from aggressive settings. Here is the a/b test result of the config changes I implemented:
- Eager end-of-turn: Switched from ON (0.35) to OFF
- VOICE_EOT_THRESHOLD: Increased from 0.5 to 0.7
- VOICE_EOT_TIMEOUT_MS: Increased from 700ms to 1200ms
The result? The "cut-ins" stopped. The response now arrives about half a second later, which feels like a natural human beat rather than a technical lag.
Analyzing the p95 Tail
When auditing your AI workflow, stop looking at p50 (median) latency. A "sub-second" p50 is a vanity metric; the p95 is what users actually experience. This is where conversations break down. My current breakdown looks like this:
- Deepgram Flux STT: ~350 ms
- VAD fallback: ~280 ms
- Cartesia Sonic TTS: ~250 ms
- Total p95: ~880 ms
To keep the experience snappy without feeling robotic, I use a specific optimization: chunking the first TTS request. Instead of buffering a full sentence, I send the first speakable fragment to Cartesia as soon as I have 4 to 8 tokens (capped at a 250ms wait). This shaves off perceived latency without triggering the "interruption" bug.
The biggest takeaway for anyone building a voice agent from scratch: distinguish between "processing latency" (the time to generate a response) and "listening patience" (the time the AI waits before speaking). Optimizing the former while ignoring the latter is a recipe for a frustrated user base.