AI Pentest Agent: From Hallucinations to Real Root Shells

LLM agents will happily report success they never actually achieved. I've been building an autonomous penetration-testing agent designed to drive tools like nmap, Metasploit, and searchsploit in a continuous loop: recon, exploit, verify, and repeat. Running this against a Metasploitable VM in an isolated lab taught me a brutal lesson in "AI honesty."

Early on, the agent handed me a flawless report: "23 of 23 ports breached, root on all." In reality, the number of shells popped was zero. The agent hadn't hacked anything; it had simply hallucinated a total compromise with absolute confidence.

The "Success" String Match Trap

The failure lived in the validator. To determine if an exploit worked, I had used a simple string match:

# The original flawed check
if "login:" in output or "shellcodes" in output.lower():
    return "confirmed"

This created two massive failure modes. First, searchsploit prints a header containing the word "Shellcodes" during every single search. The agent saw that word and flagged the port as breached before it even tried an exploit. Second, any service banner echoing "login:" was counted as a shell. The result was a report full of "High confidence" claims based on zero evidence.

To fix this, I moved from "vibes" to hard evidence. I implemented a breach_confirmed() function that requires actual proof of code execution:

_SHELL_EVIDENCE = (
    re.compile(r"uid=\d+\([a-z]+\).*gid=\d+"), # id(1) output
    re.compile(r"root@[\w.-]+:[~/]"), # a root prompt
)

def breach_confirmed(output: str) -> bool:
    return any(p.search(output) for p in _SHELL_EVIDENCE)

Once I forced the agent to prove its claims, the "23/23" success rate plummeted to 3. But those 3 were real.

Solving the Plumbing Bugs

With the lying stopped, I could finally see why the success rate was so low. It wasn't a lack of "intelligence," but boring deployment and parsing bugs:

• Schema Mismatches: The model tried to call searchsploit using {"query": "vsftpd"}, but the tool required {"keyword": "..."}. The MCP layer rejected these silently. I fixed this by allowing multiple keys (keyword/query/search).
• ANSI Poisoning: msfconsole uses color escape codes. My parser captured exploit/unix/\x1b[45mftp\x1b[0m/vsftpd_234 as the module path, which obviously failed to load. I added a strip_ansi() step to clean the logs.
• Version String Confusion: nmap sometimes reports versions first (e.g., "2 (RPC #100000)"). My parser thought "2" was the product name and sent it to Metasploit. I added a leading-digit guard to prevent this.

This journey from a "perfect" fake report to a flawed but honest AI workflow was the most valuable part of the project. Real-world LLM agent deployment is less about the prompt and more about the rigorous validation of the output.