SigNoz LLM Observability: A Deployment Guide
gen_ai.* spans and calculating pricing already exists in the open-source version (v0.133.0+), there is a significant gap between running docker compose up and actually seeing your AI data on a dashboard.The "Silent Gates" to Deployment
Most users fail to reach the LLM observability features because three specific configuration hurdles fail silently—the backend reports success, but the frontend remains empty.
1. The Observability Flag: The feature is hidden behind an experimental flag. If disabled, the UI simply redirects you to the home page without any error message.
2. Dashboard Versioning: The LLM dashboards require v2 rendering. Without the correct flag, you'll see a "Welcome to your new dashboard" shell, even if the data is flowing perfectly in the background.
3. Empty Registries: The llm_pricing_rules and span_mapper_groups registries ship empty. Without seeded prices or vendor mapping presets, the dashboards have nothing to display.
To bypass these gates and get the system operational, you must set both of these environment variables simultaneously:
SIGNOZ_FLAGGER_CONFIG_BOOLEAN_ENABLE__AI__OBSERVABILITY=true
SIGNOZ_FLAGGER_CONFIG_BOOLEAN_USE__DASHBOARD__V2=trueImproving the AI Workflow
Beyond just getting the dashboard to load, the real challenge is the lack of a "measured answer quality" signal. Most observability tools tell you that a request happened and how much it cost, but they don't tell you if the answer was actually correct.
By treating SigNoz not as a black box but as a system under test, we were able to identify that following standard documentation (like using OpenInference) often results in zero gen_ai.* attributes being emitted, leaving the mapping layer useless.
For anyone building an LLM agent or a complex AI workflow, the goal should be moving past simple "success/fail" metrics toward a deep dive into actual response quality. This requires a deliberate prompt engineering strategy to ensure your spans carry the metadata necessary for the observability layer to actually categorize and price the tokens correctly.