LLM Judges: Accuracy vs Model Size

I accidentally cited a result that didn't exist in a previous post regarding "directional failures"—where an LLM output is structurally perfect but semantically reversed (e.g., "keep" instead of "delete"). To make it right, I ran a full-scale benchmark: 20 failure scenarios across 3 model tiers, totaling 600 judgment calls.

The results show a massive performance cliff based on model scale.

Experimental Setup

I tested 20 scenarios across four categories: Explicit Directional Failures (DF), Subtle DF, Valid controls, and Garbage controls. Each scenario was run multiple times to check for consistency.

• Models used:

- qwen3:0.5b: Local (Ollama)
- gemma3:latest: Local (Ollama)
- deepseek-v4-flash: API

The judge prompt was kept strictly minimal with temperature 0 to avoid rubric bias:

Judge whether this output satisfies the task.
Output: {output}
Task: {task}
Respond in JSON: {"passes": bool, "confidence": 0-1}

Performance Breakdown

The gap between small local models and larger frontier/distilled models is jarring.

• Global Accuracy: qwen3:0.5b (61.5%) vs. gemma3/deepseek-v4-flash (both 92.0%)
• Explicit DF: qwen3:0.5b missed about 36% of direct contradictions, while the larger models hit 100%.
• Subtle DF: This is where it gets messy. qwen3:0.5b failed nearly half the time (44% miss rate). gemma3 was better but still missed ~10%. DeepSeek-v4-flash was the most robust here with only a 2% miss rate.
• Controls: Valid and garbage controls were handled well across the board, though DeepSeek oddly missed 10% of valid controls.

The Verdict

If you're building an AI workflow that relies on an LLM agent to grade other LLMs, do not use anything under 4B parameters for the judge.

Models below 1B are essentially guessing when the failure is subtle. Once you hit the 4B+ range, explicit contradictions are caught perfectly, and subtle semantic reversals become much easier to detect. For high-stakes deployment, a larger API-based model like DeepSeek-v4-flash is significantly more reliable for catching those "plausible but wrong" outputs.