Stop LLM Agents from Hallucinating API Endpoints
The scenario was a classic failure of context prioritization. I had provided the actual, up-to-date API specification within the system prompt, yet the agent continued to trigger 404 errors for three hours straight. The most frustrating part wasn't the error itself, but the model's insistence that the failure was a "temporary network glitch." It proceeded to rewrite the same broken request five times, each time with a different justification for why the endpoint should work, while completely ignoring the provided documentation.
After digging into the logs, the diagnosis was clear: the model was suffering from a conflict between its internal training data (outdated documentation) and the RAG/contextual data I provided. It was essentially choosing its "memory" over the "truth" presented in the prompt. This is a common issue when working with rapidly evolving APIs where the model has seen a previous version of the documentation during its pre-training phase.
To resolve this, I had to move away from "polite" prompting and implement a more rigid architectural constraint. I found that simply telling the model to "refer to the documentation" wasn't enough. I implemented two specific changes:
First, I shifted to a strict schema validation layer. Instead of letting the agent fire requests directly, I forced the output through a Pydantic validator (using Python 3.10+) that checks the requested endpoint against a whitelist of valid paths before the network call is even attempted. If the endpoint isn't in the whitelist, the system throws a validation error back to the LLM, forcing it to reconcile the discrepancy before it can proceed.
Second, I modified the system prompt to be far more assertive. I moved the API specifications to the very end of the prompt—utilizing the "recency bias" some models exhibit—and explicitly commanded the model to ignore any prior knowledge of the API in favor of the provided spec.
While it feels like a brute-force solution, adding a validation step between the LLM's decision and the actual execution is the only way to guarantee reliability. We cannot trust an agent to be "honest" about a 404 error; we have to build a system that makes lying computationally impossible.
If you are building agents that interact with external services, don't rely on the model's ability to "read" your docs. Implement a strict schema check and treat the LLM's output as a suggestion that requires verification before it hits your production environment.