GPT-4o’s realtime API still leaves a two-second silence—here’s how to fix it

The illusion of human-like voice AI breaks when users hit the "turn-taking gap"—a two-second pause while the system processes speech, generates text, and converts it back to audio. OpenAI’s GPT-4o Realtime API was designed to erase this delay by merging text and audio into a single model, yet developers quickly find that "low latency" requires active tuning rather than automatic optimization.

The pipeline collapse isn’t enough—timing still matters
Replacing the STT → LLM → TTS chain with a unified stream removes sequential delays, but the bottleneck shifts to handling the WebSocket connection and voice activity detection (VAD). Relying solely on OpenAI’s default server-side VAD often creates jarring interruptions—cutting off mid-breath or failing to catch the end of a user’s sentence. The solution lies in a hybrid approach: use local VAD to trigger the interrupt event immediately, bypassing the server’s slower response time.

Below 500ms latency changes everything
Once response times drop under half a second, the interaction shifts from a transactional query to a continuous dialogue. This opens doors for applications where hesitation ruins the experience—live translation, crisis support, or any scenario where a one-second lag turns fluidity into stiffness. The key? Avoid overloading the system with verbose instructions. Since the Realtime API processes audio tokens directly, lengthy prompts introduce hidden delays. Instead, set modality and voice preferences once using session.update, then let the model handle the rest.

A working configuration starts with this payload:

{
  "type": "session.update",
  "session": {
    "modalities": ["text", "audio"],
    "instructions": "Be concise. Respond in short sentences to maintain conversational flow.",
    "voice": "alloy",
    "turn_detection": {
      "type": "server_vad",
      "threshold": 0.5,
      "prefix_padding_ms": 300,
      "silence_duration_ms": 500
    }
  }
}

Interruptions must be instantaneous—or the illusion dies
The real advantage of low-latency isn’t just speed; it’s the ability to halt mid-sentence when the user speaks. If the frontend doesn’t clear the audio buffer the moment an interrupt signal arrives, the system still plays the tail end of the AI’s response, undoing all the latency work. The difference between a seamless conversation and a glitchy one now hinges on WebSocket orchestration, not just model performance.

What’s next isn’t smarter AI—it’s invisible timing. The future of voice agents won’t be about intelligence alone, but about making the stream feel effortless. Master the rhythm of these events, and the AI won’t just respond; it will react.