Flaky LLM gates are destroying our

DeepSeek-V3 and Claude 3.5 Sonnet are currently fighting a war for the "coding brain" title, but the real nightmare is using them as logic gates in an automated pipeline. I’ve spent the last month building a multi-stage agentic workflow where one model decides if a task is "complete" or "needs revision" before passing it to the next stage. The inconsistency in these "gates" is where the whole system falls apart.

When you use GPT-4o for routing, it’s generally stable but suffers from a weird "agreeableness" bias—it tends to say a task is finished even when it's 80% there just to be helpful. Claude 3.5 Sonnet is far more pedantic, which is actually what you want for a gate. It catches edge cases that GPT-4o ignores, but it’s prone to "over-correction," looping the same prompt back to the generator five times because of a trivial formatting quirk.

The real shocker is DeepSeek-V3. In raw coding benchmarks, it's a beast, but as a logic gate, it's volatile. I ran a test set of 200 complex JSON validation tasks. Claude 3.5 Sonnet hit 94% accuracy on the "Pass/Fail" decision. GPT-4o hit 89%. DeepSeek-V3 swung between 70% and 98% depending on how the system prompt was phrased. If I used a strict "Answer only YES or NO" prompt, DeepSeek sometimes hallucinated a reason why it was saying yes, which broke my regex parser.

To stop the flakiness, I had to ditch the simple prompts and move to structured output. If you aren't forcing these models into a strict schema, you're just gambling. Here is the prompt structure that actually stabilized my gates:

{
  "role": "system",
  "content": "You are a binary quality gate. Evaluate the input based on the provided rubric. 
  Return ONLY a JSON object with the keys 'status' (boolean) and 'reason' (string). 
  Do not include conversational filler."
}

Even with this, the "intelligence" of the gate varies. Gemini 1.5 Pro is a sleeper hit for long-context gates. If the "gate" needs to check a 50-page documentation file to see if a piece of code is compliant, Gemini destroys the others. It doesn't lose the thread halfway through the document, whereas Claude starts getting "distracted" by the middle of the context window (the classic lost-in-the-middle problem).

The Trade-offs:

Claude 3.5 Sonnet: Best for strict quality control and nuance, but can be too picky, leading to infinite loops in agentic flows.

GPT-4o: The most "stable" in terms of following formatting instructions, but lacks the critical eye needed for high-precision gating.

DeepSeek-V3: Incredible value and raw power, but requires significantly more prompt engineering to keep the output deterministic.

Gemini 1.5 Pro: The only viable choice for gates that require massive context windows, though its reasoning is occasionally less "sharp" than Claude's.

The biggest takeaway is that a "smarter" model isn't always a better gate. A model that is too smart might find a reason to fail a task that is "good enough" for production, whereas a dumber model might let a bug slide. The goal isn't maximum intelligence; it's maximum consistency. For now, I'm routing all my final "production-ready" checks through Claude 3.5 Sonnet because the cost of a false positive is higher than the cost of a few extra API calls.