Claude Code managed to polish our internal agent using a $10k

My manager basically handed me a $10,745 budget and a challenge: use Claude Code in a continuous loop to see if it could actually improve our enterprise AI agent's performance without us having to manually rewrite every single prompt. I've been tasked with rolling out AI across our department, and honestly, the biggest bottleneck isn't the tech—it's the endless cycle of "tweak prompt, test, fail, repeat" that eats up everyone's afternoon.

The goal was to see if an LLM agent could act as its own engineer. We set up a loop where Claude Code would analyze the agent's failure points, modify the system prompts or the underlying logic, and then run a battery of tests to see if the accuracy improved. It's essentially an automated evolution loop for prompt engineering.

The Deployment Process

Setting this up wasn't exactly "plug and play." We had to build a scaffolding that allowed the AI to actually see the results of its changes.

1. Baseline Setup: We established a gold dataset of 500 complex queries that our agent consistently struggled with.
2. The Loop: We configured a script that fed the error logs back into Claude Code. The instruction was simple: "Analyze why this failed and update the prompt to fix it without breaking existing successes."
3. Validation: Every change had to pass a regression test. If the new prompt fixed one bug but broke ten others, the loop rejected the change and tried a different architectural approach.

# This is a simplified version of how we triggered the loop
while [ $current_accuracy -lt $target_accuracy ]; do
  claude-code "Analyze logs/failures.log and optimize prompt.txt"
  npm run test-suite > results.log
  current_accuracy=$(grep "Accuracy" results.log | awk '{print $2}')
done

What Actually Happened

The cost was the scariest part. Watching the API credits burn through that $10k budget felt like watching a countdown timer. However, the results were surprising. For the first few hundred iterations, it just kept hallucinating "better" prompts that did nothing. But around the 200th loop, it started identifying patterns in the agent's reasoning errors—specifically how it handled nested JSON objects.

• Speed of Iteration: What would have taken my team three weeks of manual analysis happened in about 48 hours of compute time.
• Accuracy Gains: We saw a measurable jump in success rates for complex queries, though the gains plateaued after the first $4,000 spent.
• Pushback: My lead dev hated it at first. He argued that "spending thousands of dollars to let a bot guess the prompt" was insane. He only came around when he saw that the bot found a logic flaw in our retrieval step that we had missed for months.

It turns out that for a real-world AI workflow, sometimes the most efficient path to a "perfect" agent is just throwing enough tokens at the problem until the LLM solves its own limitations. It's not a magic bullet, but as a practical tutorial for others: if you have the budget, automating the optimization loop is way faster than manual prompt engineering.