We optimized our internal agent with a $10,745 budget using Claude Code

My manager provided a $10,745 budget and a specific challenge: run Claude Code in a continuous loop to determine if it could improve our enterprise AI agent's performance without manual prompt rewriting. My responsibility involves rolling out AI across our department, and the primary bottleneck is not the technology itself, but the exhausting cycle of tweaking prompts, testing, failing, and repeating that consumes entire afternoons.

Can an LLM agent function as its own engineer?

The objective was to test whether an LLM agent could function as its own engineer. We established a loop where Claude Code analyzed failure points, modified system prompts or underlying logic, and executed tests to measure accuracy improvements. This created an automated evolution loop for prompt engineering.

The Deployment Process

Implementation was not plug and play. We required a scaffolding that allowed the AI to observe the consequences of its modifications.

How was the baseline setup and testing implemented?

1. Baseline Setup: We created a gold dataset consisting of 500 complex queries where our agent regularly struggled.
2. The Loop: A script was configured to feed error logs back into Claude Code with a simple instruction: Analyze why this failed and update the prompt to fix it without breaking existing successes.
3. Validation: Every modification required passing a regression test. If a new prompt resolved one bug but caused ten others, the loop rejected the change and attempted a different architectural approach.

# This is a simplified version of how we triggered the loop
while [ $current_accuracy -lt $target_accuracy ]; do
  claude-code "Analyze logs/failures.log and optimize prompt.txt"
  npm run test-suite > results.log
  current_accuracy=$(grep "Accuracy" results.log | awk '{print $2}')
done

What Actually Happened

What were the costs and speed of iteration?

The expense was the most intimidating factor. Watching API credits deplete that $10k budget felt like watching a countdown timer. The results, however, were unexpected. During the initial hundreds of iterations, the system produced hallucinated prompts that yielded no improvement. Around the 200th loop, it began identifying patterns in reasoning errors, particularly regarding nested JSON objects.

• Speed of Iteration: Processes that would have required three weeks of manual analysis by my team were completed in roughly 48 hours of compute time.
• Accuracy Gains: We observed a measurable increase in success rates for complex queries, although gains plateaued after the initial $4,000 was spent.
• Pushback: My lead developer was initially opposed, arguing that spending thousands of dollars to let a bot guess prompts was irrational. He changed his stance once the bot identified a logic flaw in our retrieval step that had gone unnoticed for months.

For real-world AI workflows, the most efficient route to a perfect agent can sometimes be applying enough tokens to the problem until the LLM overcomes its own limitations. It is not a magic bullet, but as a practical lesson for others: if your budget allows, automating the optimization loop is significantly faster than manual prompt engineering.