Integrating LLMs into legacy teleph

Legacy telephony systems—think PBX, IVR, and old-school SIP trunks—are notorious for being "black boxes" that are impossible to modernize without ripping everything out. The real bottleneck isn't the voice quality; it's the rigid, decision-tree logic of traditional IVRs that frustrates every customer who doesn't want to "Press 1 for Sales."

The most efficient way to bridge this gap without a total infrastructure overhaul is by implementing an LLM-powered middleware layer using a combination of Vapi or Retell AI for the voice orchestration, and a custom FastAPI backend to handle the logic. Instead of trying to force an LLM directly into a legacy switch, you treat the legacy system as a simple audio stream provider and the LLM as the intelligent routing engine.

The core problem this solves is "intent fragility." In a standard legacy system, if a user says "I'm calling because my bill is wrong" instead of "Billing," the system fails. By piping the audio through a real-time STT (Speech-to-Text) engine like Deepgram and feeding that text into a fast model like GPT-4o-mini or Groq-powered Llama 3, you get near-instant intent recognition that can actually handle nuance.

To get this running, you don't need to rewrite your telephony stack. You just need a SIP trunk that supports WebSockets. Here is a basic conceptual flow for the backend handler to manage the hand-off from the LLM back to a legacy extension:

from fastapi import FastAPI, WebSocket
import openai

app = FastAPI()

@app.websocket("/voice-stream")
async def handle_voice(websocket: WebSocket):
    await websocket.accept()
    while True:
        # Receive transcribed text from the voice gateway
        data = await websocket.receive_json()
        user_input = data.get("text")
        
        # Use LLM to determine if the call needs to be routed to a human
        response = openai.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": "You are a telephony router. If the user needs a human, respond with [TRANSFER:DEPT_ID]. Otherwise, answer briefly."},
                {"role": "user", "content": user_input}
            ]
        )
        
        answer = response.choices[0].message.content
        
        if "[TRANSFER" in answer:
            # Send a signal to the legacy PBX to trigger a blind transfer
            dept_id = answer.split(":")[1].replace("]", "")
            await websocket.send_json({"action": "transfer", "extension": dept_id})
        else:
            await websocket.send_json({"action": "speak", "text": answer})

Is it worth the effort?

Latency is the only real enemy. If your STT -> LLM -> TTS pipeline takes more than 800ms, the conversation feels robotic and awkward. However, with the current speed of Groq or Together AI, the "uncanny valley" of voice AI is almost gone.

Cost efficiency is high. You aren't replacing hardware; you're adding a software layer. The cost of tokens for a standard support call is pennies compared to the labor cost of a human agent handling basic FAQ queries.

Deployment friction is moderate. The hardest part isn't the AI; it's the networking. You'll spend more time fighting with SIP headers and firewall rules than you will tuning the prompts. But once the bridge is built, you've essentially given a 20-year-old phone system a brain.