Optimizing Edge TTS latency for real-time voice assistants using Python

The biggest bottleneck in building a voice assistant isn't usually the LLM response time—it's the perceived latency between the AI finishing its thought and the audio actually hitting the speakers. If you're using edge-tts (the unofficial wrapper for Microsoft Edge's TTS), you'll notice that waiting for the full .mp3 file to download before playing it creates a jarring 1-2 second silence.

To get this feeling "real-time," you have to switch from file-based processing to an asynchronous stream. The key is leveraging asyncio and a playback library that can handle raw byte streams without needing a temporary file on disk.

Here is the setup that actually works for me. I use pygame for the audio mixer because it handles raw buffers better than most lightweight libraries, though pyaudio is a viable alternative if you prefer.

import asyncio
import edge_tts
import pygame

# Initialize pygame mixer for audio playback
pygame.mixer.init()

async def speak_stream(text, voice="en-US-AvaNeural"):
    # communicate() returns an async iterator of chunks
    communicate = edge_tts.Communicate(text, voice)
    
    # We use a bytearray to buffer small chunks before sending to the mixer
    # to prevent audio popping/stuttering
    buffer = bytearray()
    
    async for chunk in communicate.stream():
        if chunk["type"] == "audio":
            buffer.extend(chunk["data"])
            
            # Play chunks as soon as we have enough data to avoid gaps
            # 2KB is usually enough to keep the buffer full without lag
            if len(buffer) > 2048:
                # Note: edge-tts outputs MP3. 
                # For true zero-latency, you'd pipe this to ffmpeg 
                # to convert to raw PCM on the fly.
                # This is a simplified version:
                with open("temp_chunk.mp3", "wb") as f:
                    f.write(buffer)
                pygame.mixer.music.load("temp_chunk.mp3")
                pygame.mixer.music.play()
                while pygame.mixer.music.get_busy():
                    await asyncio.sleep(0.1)
                buffer.clear()

# For production, I wrap this in a queue system

The "Gotchas" and Performance Tweaks

The MP3 Overhead: edge-tts delivers MP3 data. The biggest productivity gain comes when you stop writing to disk entirely. If you're on Linux or Mac, pipe the stream directly into ffplay or aplay using asyncio.create_subprocess_exec. This removes the disk I/O overhead completely.

The LLM Integration: Don't wait for the LLM to finish the whole paragraph. Use a generator to yield sentences. I use a simple regex or a split on . and ? to feed sentences into the edge-tts queue. This way, the assistant starts speaking the first sentence while the LLM is still generating the third.

Configuration Tips for Better Feel:

• Voice Selection: Stick to the "Neural" voices. Some of the older ones have weird cadence shifts that make the latency feel worse than it actually is.
• Rate Adjustment: Adding a slight speed increase (e.g., +10%) often makes the assistant feel more responsive and "snappy" without sounding like a chipmunk.
• Concurrent Tasks: Run the TTS stream in a separate asyncio.Task from your LLM polling loop. If you run them sequentially, you're wasting precious milliseconds.

My current productivity stack for this:

• Cursor for the boilerplate: I use the @docs feature to index the edge-tts GitHub readmes so Cursor doesn't hallucinate old API methods.
• Claude 3.5 Sonnet for the async logic: It's significantly better at handling the async for loops and buffer management than GPT-4o, which tends to forget the await keyword in complex stream handlers.

By moving to a streaming architecture and feeding the TTS in sentence-sized chunks, I managed to drop the "Time to First Audio" from about 2.5 seconds down to roughly 600ms.