We need a harness-only benchmark because model weights aren't

Standard LLM benchmarks are basically useless for people actually building agents. We've all seen the leaderboards where a model scores high on MMLU or HumanEval, but the second you plug it into a real-world AI workflow, it falls apart. The problem is that we're measuring the "brain" (the model) but ignoring the "nervous system" (the harness).

The harness—the orchestration layer, the tool-calling loop, the error handling, and the prompt engineering—is where the actual performance wins or losses happen. A mediocre model with a brilliant harness often beats a frontier model with a lazy wrapper. We're missing a community-driven way to measure this specific variable across diverse, real-world tasks.

I'm thinking about starting a project to build a leaderboard focused entirely on harness performance. The goal would be to track how different orchestration frameworks handle complex tasks, grouped by the underlying LLM and the amount of reasoning effort (like CoT or iterative loops) being used.

To make this a legitimate deep dive, the tasks can't be synthetic. They need to be actual, complex problems pulled from open-source repos—things that actually break in production. Since this should be a community effort, the measurement criteria and the framework for the benchmark should be decided by the group, not just one person.

For those who aren't familiar, a "harness" in this context is everything that surrounds the LLM call. If you're building an LLM agent, your harness is what manages the state, parses the tool outputs, and decides when the task is actually "done."

I maintain a coding agent called Dirac, but I don't want to bake my own biases into the requirements. The idea is to create a neutral ground where we can see which architectural patterns actually work. If we can get a set of diverse tasks and a way to measure success/failure rates across different harnesses, we'd finally have a practical tutorial for how to actually build reliable agents instead of just guessing which model is "smarter."

If people are into this, I'll set up a Discord to coordinate. We could look at metrics like:

• Success rate on multi-step dependencies
• Token efficiency per successful task completion
• Recovery rate after tool execution errors
• Latency overhead added by the orchestration layer

This would be a massive step toward a more real-world understanding of deployment. Instead of arguing about whether Claude 3.5 or GPT-4o is better at a specific benchmark, we could see which harness implementation actually ships the code.