SigNoz + OpenTelemetry: Building an AI Agent Error Budget Gateway

Giving an AI agent write access to a database is a gamble unless you have a mechanical way to revoke those permissions the second things go sideways. "Human-in-the-loop" is a bottleneck that doesn't scale at 3 AM, so I built LEASH—a system that treats agent permissions like an SRE error budget. If the agent burns through its budget of allowed failures, its authority is automatically throttled.

The Architecture of LEASH

Instead of a single monolithic script, I decoupled the agent from the policy enforcement to ensure the "leash" is a separate process. The system is split into four components:

• agent-runner: The LLM agent executing tasks like apply_migration or delete_staging_table.
• leash-broker: The gateway. Every tool call must pass through here. It checks the current permission tier and handles webhooks from SigNoz.
• migration-tool & resource-tool: The actual downstream services performing the work.

I implemented a three-tier authority system:

• T3: Full authority (Read/Write/Destructive).
• T2: Restricted access.
• T1: Read-only. Destructive actions are hard-blocked.

Implementing the AI Workflow with OpenTelemetry

The core of this setup is a real-world deployment of the OpenTelemetry Python SDK. Everything ships spans, metrics, and logs to SigNoz via OTLP.

The critical detail here isn't the policy logic, but the telemetry metadata. I configured specific span names to ensure the traces are self-documenting:

• leash.policy.decision: Stores the current tier, required tier, and the final verdict.
• leash.tool.execute: Captures the actual tool call.

By doing this, the trace becomes the audit log. You don't need to grep through text files to find out why a migration was blocked; the span attributes in SigNoz provide the evidence.

Automated Demotion via SigNoz Alerts

To make this autonomous, I bypassed manual monitoring and used SigNoz metric alerts to trigger the permission changes.

I configured a metric alert on leash_tool_calls_total, specifically filtering for tool_name = apply_migration and outcome = error. When the failure rate exceeds a set threshold within a 5-minute sliding window, SigNoz fires a webhook to the leash-broker, which instantly demotes the agent's tier.

This transforms the agent from a potential liability into a managed service with a programmable safety net.