Stop trusting raw benchmark scores without looking at the harness
This is the fundamental crisis currently hitting the ARC-AGI-3 benchmarks. On July 24, the official ARC Prize verified Claude Opus 5 at 30.16%. By late August, NVIDIA reported that same model hitting 100.00% on the same set. The model didn't evolve in a month; the code wrapped around it did.
The massive gap in the data
When we look at the ARC-AGI-3 public set, the discrepancy between the "official" way of testing and the "optimized" way of testing is staggering. Here is how the numbers actually shake out across different implementations:
- Official ARC Prize harness: 30.16% (Verified)
- OpenAI (Default settings): 13.3% (Unverified)
- OpenAI (With reasoning retention/compaction): 38.3% (Unverified)
- Impossible Research (Schema): 98.98% (Unverified)
- MIT (VISTA): 100.00% (Unverified)
- NVIDIA (AVO): 100.00% (Unverified)
The "real" delta here is roughly 70 points. We are seeing a massive spread where the official harness is designed to be intentionally generic—almost punishingly so—to expose model weaknesses. Meanwhile, the high-scoring harnesses are built to augment the model, essentially covering its shortcomings through clever engineering.
Why the "official" score is often a floor, not a ceiling
The official harness uses a rolling truncation window. As the conversation history grows, older actions and reasoning steps are simply deleted to save space. In practice, this means the harness is effectively wiping the model's "short-term memory" between moves.
When OpenAI applied compaction and reasoning retention, they didn't change the LLM; they just stopped the harness from lobotomizing the model mid-task. That single change tripled the score. This highlights a massive problem in prompt engineering and LLM agent deployment: the environment is just as important as the weights.
The move toward training through the harness
The trend is getting even more aggressive. Microsoft recently released a framework where the harness is actually inside the training loop. This is a sophisticated form of reward hacking. If you use Reinforcement Learning (RL) where the deploy-time harness owns the reward loop, the harness itself becomes part of the model's learned behavior.
We are moving into an era where a benchmark score is a "self-reported claim" unless it includes:
1. The specific harness version used.
2. The exact memory state management.
3. The action budget allowed per task.
If you're building an AI workflow or an LLM agent, stop looking at the raw percentage on a leaderboard. A 100% score on a specialized harness might actually perform worse in a real-world, unoptimized environment than a 30% model running on a clean, generic baseline. The benchmark is no longer a measure of intelligence; it's a measure of how well the agent and its environment are synchronized.