Weir
So I built Weir — deterministic unit tests for AI agents that don't need an LLM to run. It chews on the OpenTelemetry traces you're probably already collecting and answers two questions:
- How much of your agent's behavior your telemetry can actually prove happened
- Whether a forbidden flow slipped through, shown as a witness path you can fail CI on
Same input, byte-identical answer every time. Nothing leaves your machine. No LLM in the loop.
This is the kind of thing I'd slap into our CI pipeline at work and watch the compliance team's blood pressure drop. Let's be real — most of us are flying blind on what our agents actually do once they hit production. Logs say "done" but who's verifying the how?
The install is stupid simple if you want to poke at it:
pip install weir-scan && weir gauge --sampleIt's Apache-2.0 licensed, and the roadmap is totally out in the open. I'd genuinely love to know where it breaks on your trace exports — there's a GitHub repo floating around but I'll let you hunt for it.
Used Claude Code to help draft this post, which honestly tracks. The tool itself? Built to make sure your agent doesn't become a black box with a success stamp.
Your experience with testing agents sounds like it could save others some serious headaches — what's one lesson you wish you'd learned earlier in the process?