Implementing Low-Latency Streaming TTS for Real-Time AI Voice Agents

Getting an AI voice agent to feel "human" isn't about the quality of the voice—it's about the gap between the LLM finishing a sentence and the audio hitting the speaker. If you're hitting 2-3 seconds of latency, the illusion is broken. I've spent the last month optimizing a streaming pipeline using Cursor and Claude 3.5 Sonnet, and the biggest win was moving from "request-response" to a true chunked stream.

The architecture that actually works involves a three-way pipe: LLM (streaming text) → TTS Engine (streaming audio chunks) → Frontend (playing audio buffer).

Most people make the mistake of waiting for the LLM to finish a full sentence before sending it to the TTS. Instead, I use a regex-based buffer that flushes text to the TTS API the moment a punctuation mark (,, ., ?, !) appears. This allows the TTS to start synthesizing the first clause while the LLM is still generating the second.

Here is the core logic I implemented to handle the text chunking:

class TTSStreamBuffer {
  private buffer = "";
  private delimiters = /[,\.!\?\n]/;

  push(text: string, onFlush: (chunk: string) => void) {
    this.buffer += text;
    const match = this.buffer.match(this.delimiters);
    
    if (match) {
      const splitIndex = match.index! + 1;
      const toFlush = this.buffer.slice(0, splitIndex);
      this.buffer = this.buffer.slice(splitIndex);
      onFlush(toFlush);
    }
  }
}

For the TTS engine, I switched from standard REST calls to WebSockets. Using a provider like ElevenLabs or Deepgram via WebSocket reduces the HTTP overhead significantly. The trick is to send the text frame and immediately listen for audio frames.

One major "gotcha" I encountered: audio popping. When you stream chunks, the edges of the audio buffers often don't align perfectly, causing a clicking sound. To fix this, I had to implement a small cross-fade or ensure I was using a player that handles PCM data smoothly.

If you're using Cursor, I highly recommend creating a .cursorrules file specifically for your audio pipeline. I told mine to "always prefer TypedArrays for audio binary data and avoid any blocking synchronous calls in the audio playback loop." This stopped the AI from suggesting fs.readFileSync in my Node.js backend, which was killing my event loop.

My current production stack for < 800ms latency:
LLM: GPT-4o-mini (for speed) with stream: true
TTS: Deepgram Aura (insanely fast TTFB)
Transport: WebSockets for both LLM-to-Server and Server-to-Client
Frontend: Web Audio API using an AudioWorklet to queue and play raw PCM chunks

Another productivity tip: when debugging latency, don't trust your "feel." I added a simple timestamp log to every packet: [LLM_GEN] -> [TTS_REQ] -> [TTS_RESP] -> [CLIENT_PLAY]. It turned out my frontend was buffering too much audio before starting playback, which added 400ms of unnecessary lag. Reducing the client-side buffer size to just two chunks fixed it.