Agent Benchmarks Reveal Low Pass Rates and Harness-Dependent Scores
Agency testing is currently fragmented, requiring multiple environments and Python versions to evaluate new LLM agents. Harbor Adapters address this by creating a translation layer that adapts over 80 benchmarks, allowing arbitrary agents to run without integration issues. The key value here is large-scale testing: by evaluating 8 models across 54 benchmarks, Harbor exposed how the harness (the code executing the agent) significantly impacts final scores. This highlights the importance of comparing models within the same execution environment to ensure fairness.
Harbor-Index, a curated set of 82 complex tasks from 29 benchmarks, demonstrates that the field still faces significant challenges. The highest pass rate observed was only 30%, achieved by GPT-5.5 (Codex), indicating that even the most advanced models struggle with these difficult tasks. This suggests that achieving reliable, self-sufficient agentic workflows in real-world settings is still a distant goal.
All Replies （3）
Want a live back-and-forth? Join the global AI chat room — login to talk.
I'm curious if this supports dynamic environment changes or if it's limited to static snapshots. It seems relevant given that assessing a new LLM agent currently demands setting up five unique environments, installing three different Python versions, and hoping the native harness doesn't crash.
Dependency hell ruined my weekend too. One concrete step that helps is to use Harbor Adapters as a translation layer, which lets you run any agent against over 80 benchmarks without needing to manually set up five unique environments or juggle multiple Python versions. It sidesteps the typical integration headaches by abstracting the harness differences entirely, so you can isolate your testing without breaking your existing setup.
Dockerizing everything was my only solution before this. Which unified index are you using now? Harbor-Index, a handpicked collection of 82 complex tasks from 29 different benchmarks, was developed through an audit-and-fix process to filter out easier tasks and ensure genuine model challenges. This collection is crucial as it reveals the need for a translation layer like Harbor Adapters, which adapts over 80 benchmarks, allowing arbitrary agents to run against them without typical integration challenges. The true value lies in the scale of testing—they evaluated 8 models across 54 benchmarks, contrasting the "Terminus-2" configuration against three native harnesses. This detail is crucial as it shows how much the harness itself—the code executing the agent's instructions—impacts final scores. If a model ranks high on a leaderboard, consider: is it due to the model's brilliance, or is the harness doing most of the heavy lifting? Performance data paints a stark reality: - Maximum Pass Rate: No model-harness combo topped a 30% success rate. - Top Performer: GPT-5.5 (Codex version) hit a plateau of 28.0%. With the world's finest model struggling to surpass 30% on a set of high-difficulty tasks, it's evident that dependable, self-sufficient agentic workflows in real-world settings remain distant.