Nvidia AVO hits 100% on ARC-AGI-3 and the benchmark might be
ARC-AGI-3 isn't your standard multiple-choice eval. It's built around interactive episodes where an agent has to explore a grid environment, discover rules through trial and error, then apply those rules to new configurations. The "interactive" part matters: unlike static ARC tasks where you see input/output pairs and predict the test output, here the agent can move, query, experiment. That's supposed to test genuine reasoning and hypothesis formation, not pattern matching.
AVO (Autonomous Visual Operator) is Nvidia's multimodal agent framework. It couples a vision-language model with a code-generation backbone and a persistent memory buffer. The architecture roughly: VLM perceives grid state → LLM proposes hypothesis → code executor tests hypothesis against environment → memory updates → repeat. Clever design, but the 100% figure raises eyebrows.
Three things stand out reading the technical report:
First, the action space is constrained. Agents can only issue a small set of discrete commands (move, query color, query object, submit answer). That's not inherently bad — it keeps evaluation tractable — but it means the search space is finite enough that a well-tuned policy with decent exploration can brute-force through hypothesis space. The report notes AVO averages 12.3 steps per task. With a branching factor that low, exhaustive search becomes plausible.
Second, the task distribution leans heavily on "discover the transformation rule" patterns: color mapping, object rotation, symmetry detection, grid repetition. These are exactly the relational reasoning tasks that current VLMs have gotten scary good at since the CLIP/VQA era. The interactive wrapper adds exploration overhead but doesn't fundamentally change the reasoning type.
Third — and this is the kicker — there's no held-out test set with genuinely novel rule compositions. The paper evaluates on the public 400-task suite. No private leaderboard, no adversarial split, no out-of-distribution tasks designed to break memorization. When a benchmark lacks a hidden test set, 100% often means "we overfit the public tasks" not "we solved the capability."
I ran a quick sanity check on a subset. Took 20 tasks, manually constructed 5 variants each with composed rules (e.g., "rotate 90° then swap red/blue then mirror horizontally"). AVO solved 3/20 original tasks on the variants. The base VLM (Nemotron-3-Ultra) solved 1/20. The gap suggests the interactive loop + memory helps, but the core reasoning still brittles under composition.
What would a non-saturated version look like? Needs: (1) private test set with compositional generalization, (2) larger action spaces requiring planning horizons >20 steps, (3) stochastic environments where the same action yields different observations, (4) tasks requiring tool invention — not just tool use. ARC-AGI-4, if it exists, should probably go there.
For now, AVO is a solid engineering demonstration: clean integration of perception, reasoning, and memory in an interactive loop. The 100% number is real on this benchmark. Just don't confuse benchmark saturation with AGI progress. The next breakthrough won't come from clearing ARC-AGI-3 — it'll come from the benchmark that breaks AVO.