Implementing Real-time Multimodal Agents using Gemini 2.0 Flash and WebSockets

The Gemini 2.0 Flash Multimodal Live API is a game-changer because it kills the "turn-based" latency that makes most voice bots feel robotic. Instead of the old flow (Audio → STT → LLM → TTS → Audio), you're piping raw audio/video frames directly into the model via WebSockets and getting a real-time audio stream back.

I've been building a prototype for a "live coding tutor" that watches my screen and talks me through bugs, and the biggest hurdle isn't the AI—it's managing the binary frames over the socket.

To get this running, you need to move away from standard REST calls. The Multimodal Live API requires a persistent WebSocket connection to wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.MultimodalLiveService.

Here is the core logic for handling the setup and sending a configuration message to trigger the "Live" mode:

const setupLiveSession = async (socket) => {
  // The setup message is mandatory to define the model and system instructions
  const setupMessage = {
    setup: {
      model: "models/gemini-2.0-flash-exp",
      generation_config: {
        response_modalities: ["AUDIO"], // Force audio output for low latency
        speech_config: {
          voice_config: { prebuilt_voice_config: { voice_name: "Aoede" } }
        }
      },
      system_instruction: {
        parts: [{ text: "You are a real-time technical assistant. Keep responses concise and conversational." }]
      }
    }
  };
  socket.send(JSON.stringify(setupMessage));
};

The real "productivity gain" here is using the realtime_input event. If you're building a web app, you can't just send a giant file; you have to stream PCM audio chunks (16kHz, mono, little-endian).

Crucial Implementation Gotchas:

Audio Chunking: Don't send audio packets too frequently or too slowly. I found that sending chunks every 100ms-200ms provides the best balance between perceived latency and socket stability.

The "Interruption" Problem: This is the hardest part. The model doesn't automatically stop talking just because the user started speaking. You have to implement a client-side Voice Activity Detection (VAD) system. When VAD triggers, you must send a client_content message with an empty turn or a specific signal to "interrupt" the model's current output stream, otherwise, the AI will keep talking over the user.

Handling Binary Data: The API returns audio as base64 encoded strings within JSON messages. If you try to decode these on the main UI thread in a browser, you'll see micro-stutters. Move the audio decoding and playback to a Web Worker or use the Web Audio API's AudioWorklet for a glitch-free experience.

For the video side, I'm currently sending base64 encoded JPEG frames at 1fps. Gemini 2.0 Flash is surprisingly good at understanding context from these snapshots without needing a full 30fps stream, which saves a massive amount of bandwidth.

// Example of sending a screen frame
const sendFrame = (socket, base64Image) => {
  const frameMessage = {
    realtime_input: {
      media_chunks: [{
        mime_type: "image/jpeg",
        data: base64Image
      }]
    }
  };
  socket.send(JSON.stringify(frameMessage));
};

If you're coming from a GPT-4o Realtime API background, the Gemini implementation feels more flexible regarding modality mixing, but requires a bit more manual lifting on the frontend to handle the audio buffer correctly. The latency is effectively sub-second, making it the first time an AI agent actually feels like it's "in the room" with me.