Observability is the only way to stop guessing why your AI

The team built a system where five different open-weight models (GLM, Qwen, Kimi) handle planning, frontend, backend, reviewing, and self-repair. Instead of just logging outputs, they instrumented everything with OpenTelemetry. The result? The telemetry proved their initial assumptions about the system were almost entirely wrong. They were blaming models for errors that were actually caused by their own hard-coded limits or a "strong" review agent that was actually the weakest link in the chain.

If you're building LLM agents, this is a huge reminder: reading the code isn't enough. You need actual spans and traces to see where the logic is drifting.

The DevSwarm Agent Architecture

The swarm splits tasks across specialized open-weight models to maximize efficiency:

• Planner (GLM-5.2): Converts prompts into typed build plans and API contracts.
• Frontend (GLM-5.2): Generates a self-contained index.html.
• Backend (Qwen3-Coder-480B): Builds an Express server based on the contract.
• Critic (Kimi-K2.7-Code): Reviews both for security and bugs; gates the merge.
• Doctor (GLM-5.2): Analyzes the swarm's own traces to repair model routing.

The "Critic" is the most critical part of the AI workflow. It acts as a quality gate, routing errors back to the specific agent that caused them for a patch. If it fails after two rounds, it ships with a verdict.

Implementing Telemetry from Scratch

The team prioritized observability before polishing the code because multi-agent systems fail silently. A request might "succeed" technically, but produce a broken artifact, or a fallback might kick in so seamlessly that you don't realize your primary model is down.

To get this running, they used a self-hosted SigNoz setup. For those looking for a practical tutorial on deployment, they managed their configuration via a casting.yaml file to ensure the entire observability stack was reproducible.

They tracked every model call as a span named llm., following GenAI semantic conventions. This allowed them to move from "I think the model is hallucinating" to "I can see the exact span where the latency tripled and the token count spiked," making the debugging process data-driven rather than anecdotal.