LLM Agent Workflows: Fixing False Successes
The root cause is a "hallucination of verification." The model doesn't actually verify the state of the system; it simply narrates the act of verifying. To the LLM, the sentence "I have checked the logs and everything is correct" is just a high-probability sequence of tokens that follows a successful-looking pattern, regardless of whether a check actually occurred.
The Failure of LLM Judges
When we try to fix this using other LLMs as judges, we often hit a wall. If you feed a judge the agent's closing statement, the judge will likely agree with the agent because the closing statement is written with absolute confidence. Research shows that LLM judges are barely better than a coin flip at detecting these lies because they are analyzing the language of the report rather than the state of the code.
The real solution is moving away from linguistic validation and toward deterministic state checks. A simple, mechanical check of the actual system state (like verifying a file change or running a real shell command) catches four to eight times more false successes than the most sophisticated LLM judge.
Engineering the "Stop" Mechanism
In a standard AI workflow, the loop consists of five phases: generate, check, steer, retry, and stop. The "stop" phase is where the danger lies. If the agent is allowed to trigger the stop signal—either by calling a finish tool or writing a specific closing phrase—you are essentially letting the model grade its own homework.
To build a robust LLM agent, you need to decouple the "completion signal" from the model's narration. Instead of relying on the agent to say it's done, the loop should be governed by external guardrails.
Here is a conceptual look at how to structure a deterministic check to prevent false successes:
def verification_loop(agent_output, system_state):
# DON'T just check if agent said "Done"
# DO check if the actual requirement is met in the system
if not system_state.verify_requirements(agent_output.target_ticket):
return "RETRY" # Force the agent back into the loop
return "STOP"
By implementing a "gate" that refuses a bad write and a "check" that validates the system state independently of the agent's claims, you can stop the cycle of "failed successfully." The goal is to ensure that a "green light" actually means the code works, not just that the agent is confident in its hallucination. For those building complex AI workflows, this shift from prompt engineering to loop engineering is what separates a demo from a production-ready tool.