PIMiner can crack Gemini-2.5-Pro with a 76% success rate

Most red-teaming for prompt injection relies on reinforcement learning, but the problem is that RL models are usually too specialized. You train an attacker to break one specific LLM, and the second you switch targets, the attack efficiency plummets. It's a tedious cycle of retraining that doesn't scale when new models drop every few weeks.

The PIMiner approach flips this by using an agentic system to build a "strategy library" instead of just a static weights-based attacker. Instead of being locked into one target, it learns from a variety of dataset and model pairs. This means it's actually building a repertoire of attack patterns that it can transfer to a totally unseen LLM without needing another training round.

What's actually impressive here is the efficiency. We aren't talking about thousands of brute-force attempts to find a hole in the system. PIMiner only needs about 10 queries per sample to figure out how to bypass the target agent. It's basically a highly efficient AI workflow for finding vulnerabilities.

The numbers from the benchmarks are pretty eye-opening:

• IPIArena ASR: Gemini-2.5-Pro hit 76.2%, GPT-5.1 hit 61.9%, and Claude-Sonnet-4.5 was the toughest at 42.9%.
• AgentDojo ASR: Gemini-2.5-Pro jumped to 86.7%, GPT-5.1 sat at 53.3%, and Claude-Sonnet-4.5 stayed around 40.0%.

From a security perspective, this confirms that "agentic" red-teaming is way more dangerous (and useful) than standard prompt engineering. When you have an LLM agent whose only job is to analyze the target's responses and pivot its strategy in real-time, the defense has a much harder time keeping up.

If you're building an LLM agent and relying on basic system prompts for security, this is a wake-up call. The gap between the "strongest" models like Claude and the others is still visible, but none of them are bulletproof against a systematic, strategy-driven attack. For anyone doing a deep dive into LLM security, shifting from static test sets to these kinds of dynamic agentic attackers is the only way to get a real-world sense of how fragile your deployment actually is.