Nights Watch: A Runtime Resilience Layer for AI Agents

Agent failures are rarely dramatic crashes; they are usually "silent drifts." An agent tasked with finding a flight under $400 might accidentally book a $1,200 ticket and report "done." Technically, the task finished, but the outcome is a disaster. This is why I developed Nights Watch—a system designed to catch an agent drifting off-course in real-time and automatically trigger a recovery.

The Architecture of Resilience

The core philosophy here is that the safety net cannot be a point of failure. If your resilience system depends on an external API to perform a rollback, you've just added another way for the system to break.

I structured the deployment using a strict split:

• Local Critical Path (SQLite): I used Node's node:sqlite for a Checkpoint Manager. Every agent step—including the plan, budget consumed, and completed actions—is written as a durable checkpoint to disk. Rollbacks pull from here exclusively.
• Decision Support (SigNoz): I integrated SigNoz for context and explanation. The Policy Engine queries the SigNoz Query API for prior-run context to score severity, and the Explanation Layer uses the SigNoz MCP server to ground natural-language explanations in actual trace data. If SigNoz is unreachable, the system degrades gracefully rather than crashing.

Turning Observability into Action

Most people treat OpenTelemetry as a post-mortem tool, but for a real-world AI workflow, it needs to be load-bearing. I implemented this in three ways:

1. Correlated Tracing
Instead of simple logs, every meaningful event is a child span under a parent run.execute span. This ensures a complete audit trail of the agent's logic.

const SpanNames = {
 RUN_EXECUTE: "run.execute",
 EXECUTOR_STEP: "executor.step",
 CHECKPOINT_CREATED: "checkpoint.created",
 POLICY_EVALUATION: "policy.evaluation",
 POLICY_EXPLANATION: "policy.explanation",
 CHECKPOINT_RESTORED: "checkpoint.restored",
} as const;

2. Mid-run Context Queries
The Policy Engine doesn't judge steps in isolation. It queries SigNoz during the run to see if previous scores have already crossed a pause threshold, escalating the severity if the agent is repeatedly failing.

if (prior.queried && prior.previousScores.some((s) => s >= config.pauseThreshold)) {
 // escalate — this isn't the agent's first warning this run
}

3. Dynamic Explanations via MCP
When a policy violation occurs, the system performs a JSON-RPC tools/call to signoz_search_traces. This replaces canned error messages with actual data from the trace.

[signoz-mcp] INVOKING tools/call signoz_search_traces run=run-872ac2d2 endpoint=http://localhost:8000/mcp

By shifting from "monitoring after the fact" to "intervening during execution," we can actually deploy LLM agents in production without worrying about them quietly spending the entire budget on the wrong task.