Tutorial

My manager asked me to "integrate AI agents into our workflow" and I said sure, because what's the worst that could happen. Three months later, I had four agents running a trading operation — one pulling Nifty option chain data, one crunching PCR and open interest, one running XGBoost predictions, and one blasting Telegram alerts. My P&L was up ₹96,000 over six months, but I had no idea whether those agents earned their money or I was just manually overriding their mistakes.

So I built an agent evaluation harness — a systematic framework for stress-testing AI agents before trusting them with real money. And I discovered that almost everyone building agents online skips evaluation entirely. They build something, run it twice, see it work, and ship it. Then they're surprised when it breaks in production.

Here's what I learned, and what most people get wrong.

What an Agent Evaluation Harness Actually Does

An evaluation harness answers one deceptively simple question: how good is this agent, really? It forces you to define success metrics, build test suites covering real scenarios, run repeated evaluations, measure regressions after every change, and track whether version 2 is actually better than version 1. This isn't optional — it's engineering 101, and it's what separates a prototype from something you can trust.

Five Mistakes I Found in Every Framework I Reviewed

I looked at 50+ "agent evaluation" guides and kept finding the same blind spots:

• Single-task testing. People test one scenario — "Can it book a flight?" — and call it done. Real agents face thousands of variations. "Book a flight from Delhi to Mumbai on Friday" is a completely different prompt than "Book a flight from Delhi to Mumbai next Friday" or "Book a flight from Delhi to Mumbai on August 15th." You need variation, not one lucky example.
• Happy-path-only testing. Nobody tests failure modes. What happens when the airport name is misspelled? When the date is ambiguous? When the preferred airline isn't available? A good harness tests the messy middle, not the clean success path.
• No baseline. Reporting "85% accuracy" means nothing without context. Eighty-five percent compared to what — a random guess, a human operator, the previous version of your agent? Always compare against something.
• No regression testing. Test once, ship, never look back. Every code change can silently break an agent. Continuous evaluation catches regressions before they hit production.
• Ignoring cost. An agent that's 95% accurate but costs ₹10 per query is worse than one that's 85% accurate at ₹0.01 per query. Measure both accuracy and cost, or you'll bleed money on clever models nobody can afford to run.

My Harness Architecture

I built this for local AI agents, meaning everything runs on my machine — no cloud API calls, no external dependencies, full control. The harness has five components:

┌─────────────────────────────────────────┐
│ Agent Evaluation Harness │
├─────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌──────────────┐ │
│ │ Test Suite │ │ Runner │ │
│ │ Generator │ │ Engine │ │
│ └─────────────┘ └──────────────┘ │
│ ↓ ↓ │
│ ┌─────────────┐ ┌──────────────┐ │
│ │ Agent │ │ Metrics │ │
│ │ Under │ │ Collector │ │
│ │ Test │ │ │ │
│ └─────────────┘ └──────────────┘ │
│ ↓ ↓ │
│ ┌─────────────┐ ┌──────────────┐ │
│ │ Baseline │ │ Report │ │
│ │ Comparator │ │ Generator │ │
│ └─────────────┘ └──────────────┘ │
│ │
└─────────────────────────────────────────┘

The Test Suite Generator creates test cases automatically — for my trading agents, I generate variations across symbols, date ranges, market conditions, and edge cases like missing data or malformed inputs. The Runner Engine executes each test case against the agent under evaluation and captures outputs. The Metrics Collector tracks accuracy, latency, error rate, and token cost per run. The Baseline Comparator runs the same tests against a reference agent or a random-chance model so you know if your latest change actually improved anything. The Report Generator produces a summary so I can see at a glance whether the agent is regressing or improving.

The biggest shift for me was realizing that evaluation isn't a one-time checklist — it's a continuous loop. Every time I tweak a prompt, swap a model, or change a data source, the harness runs the full suite and tells me whether I made things better or worse. It turned agent development from guesswork into actual engineering.

If you're building local AI agents — whether for trading, customer support, or anything else — stop shipping without evaluation. The harness isn't complicated to set up, but it's the difference between an agent you trust and an agent you're secretly babysitting.