OpenTelemetry + SigNoz: Tracing Streaming Voice AI

Standard APM tools are practically useless for Voice AI because they can't distinguish between a slow STT transcription, LLM Time to First Token (TTFT), or TTS synthesis lag. In a typical CRUD app, you just trace a request-response cycle; in Voice AI, you're dealing with async generators where the span often ends when the generator is created, not when the audio actually hits the user's ears.

To solve this, I built a custom instrumentation layer to track the full "Voice Turn" lifecycle and visualize the bottlenecks in SigNoz.

The Streaming Challenge

A voice interaction is a complex chain: Raw Audio → STT → LLM Stream → TTS → Audio Stream. If a user experiences a 3-second delay, you need to know exactly which segment of that pipeline is failing. Because LLMs return async generators, auto-instrumentation fails to capture the actual duration of the stream.

Step-by-step Implementation

I implemented a custom Python decorator to manually manage the OpenTelemetry tracer. Instead of simply wrapping the function, it injects a context object that allows the AI logic to report specific milestones (like the exact moment the first audio byte is generated).

from opentelemetry import trace
import time

tracer = trace.get_tracer("zooid.voice")

def instrument_voice_app(func):
 async def wrapper(*args, **kwargs):
 with tracer.start_as_current_span("voice_turn") as span:
 turn_start = time.time()
 context = VoiceTurnContext()
 
 try:
 # Execute the Voice Agent
 result = await func(context, *args, **kwargs)
 
 # Record Voice-Specific Metrics
 if context.first_audio_time:
 ttfa = context.first_audio_time - turn_start
 span.set_attribute("voice.ttfa_ms", ttfa * 1000)
 
 if context.stt_confidence:
 span.set_attribute("stt.confidence", context.stt_confidence)
 
 if context.cost_usd:
 span.set_attribute("voice.turn_cost_usd", context.cost_usd)
 
 span.set_status(trace.StatusCode.OK)
 return result
 
 except Exception as e:
 span.set_status(trace.StatusCode.ERROR, str(e))
 span.record_exception(e)
 raise
 return wrapper

The SigNoz Visualization Hurdle

While the traces appeared in the SigNoz "Traces" tab, I hit a wall when trying to build dashboards for voice.ttfa_ms or token costs. The dashboard showed "No Data."

The issue lies in how SigNoz handles attributes in ClickHouse. When span.set_attribute sends a number, it's stored in the trace schema, but the custom dashboard query builder relies on materialized views to map these attributes. If the attribute isn't indexed correctly in the underlying ClickHouse table, the time-series panels won't pick it up, even though the data exists in the individual trace.

This setup transforms the debugging process from "the voice bot feels slow" to "the TTS provider is adding 800ms of latency," making it a practical tutorial for anyone deploying real-world LLM agents.