Harbor-Index proves that most agent benchmarks are actually too

The biggest headache with agentic evaluation right now is the fragmentation. If you want to test a new LLM agent, you usually have to set up five different environments, install three different versions of Python, and pray the native harness doesn't crash. This is exactly why the Harbor Adapters project is interesting—it basically creates a translation layer that ports over 80 different benchmarks so you can run arbitrary agents against them without the usual integration nightmare.

The scale of the testing here is what actually matters. They ran 8 different models across 54 benchmarks, comparing the results when using the "Terminus-2" setup versus three different native harnesses. This is a critical detail because it highlights how much the "harness" (the wrapper that executes the agent's code) affects the final score. If you're seeing a model perform well on a leaderboard, you have to ask if it's because the model is smart or if the harness is just doing the heavy lifting for it.

The Harbor-Index benchmark results

The most useful output here is the Harbor-Index, which is a curated subset of 82 high-difficulty tasks pulled from 29 different benchmarks. They basically filtered out the "easy" stuff using an audit-and-fix loop to ensure the tasks actually challenge the models.

The performance numbers are a reality check for the current state of AI agents:

• Maximum Pass Rate: No model-harness configuration managed to break a 30% success rate.
• Top Performer: GPT-5.5 (using Codex) hit a ceiling of 28.0%.

When the strongest model in the world can't even clear 30% on a curated set of "hard" tasks, it shows that we are still very far from reliable, autonomous agentic workflows in real-world scenarios.

How the infrastructure works

For anyone trying to build a custom evaluation pipeline from scratch, the Harbor Adapters approach is the way to go. Instead of writing a new integration for every single benchmark, they've standardized the interface.

If you are looking to implement a similar AI workflow for testing, the logic follows this pattern:
1. Adapter Layer: Create a wrapper that maps the agent's output (e.g., a tool call or a bash command) to the specific requirements of the benchmark environment.
2. Parity Testing: Run the same prompt through both the native harness and the adapter to ensure the results are identical.
3. Difficulty Filtering: Use a "difficulty filter" to remove tasks where the pass rate is too high, leaving only the edge cases that actually test reasoning.

This setup allows for a much deeper dive into failure modes. Instead of just seeing a "Fail" grade, you can analyze whether the agent failed because of a logic error or because the adapter failed to execute the command correctly.

The fact that they are open-sourcing these adapters means we can finally stop relying on the "cherry-picked" results often found in model release notes and start running a real-world, standardized battery of tests across different LLM agents.