Multi-agent workflows: Why more agents often mean worse results

I spent a huge chunk of my time building agentic systems based on the "industry standard" logic: a planner makes things better, and a panel of drafters with a judge makes them even better. It sounds logically sound—more thinking and more review should equal better answers. I didn't actually measure it until I built a tool to test my own setup, and the results were a slap in the face.

The reality check

I ran twenty coding tasks across three different harness shapes. The results were brutal:

• One drafter: 95% score | 1 call/task | $0.031 cost | 2.2s latency
• Planner → Drafter: 90% score | 2 calls/task | $0.264 cost | 9.1s latency
• Planner → Two Drafters → Judge: 80% score | 4 calls/task | $0.692 cost | 18.3s latency

The complex scaffolding actually performed worse while costing 22x more. The "panel" approach didn't beat the single drafter on a single task out of twenty; in fact, it lost three. There were no errors or crashes—it just produced inferior work, slower and way more expensively.

The danger of small eval suites

The tool gave me a critical insight: 95% vs 80% looks decisive, but it isn't. Seventeen of those twenty tasks were ties. Only three tasks actually had different results. In statistics, three discordant tasks aren't enough to prove the panel is "worse," but they are certainly enough to prove there's no evidence the extra spend is buying anything.

This is where most people mess up their AI workflow. Small test suites produce "confident nonsense." If your suite is too small, a single noise-driven failure looks like a massive regression. The only real fix is increasing the number of tasks to improve resolution.

Technical breakdown of the harness

To get these numbers, I treated the harness config as data—roles, models, prompts, and a topology graph. You can define the shape via JSON, declare the axes you want to vary, and run the matrix.

For the scoring, I kept it strictly deterministic:
1. The system generates code.
2. Predicates execute that code.
3. A binary verdict is returned.

I specifically ensured no model grades the output. This removes "judge variance" where a model might grade the same answer differently on different days.

For the comparison, I used a paired per-task approach rather than comparing averages. This is an exact sign test—no assumptions about normality or variance, and ties are excluded since they provide no directional data.

One quick deployment tip: if you're making browser-direct calls, watch out for CORS. api.openai.com can be tricky with access-control-allow-origin headers on the actual response, which can cause the browser to discard valid calls.

If you're building an LLM agent, stop assuming more agents = better quality. Measure the actual delta before you scale the complexity.