LLM Safety: Why ASR is a Bad Metric for Defenders

Attack Success Rate (ASR) is the industry standard for measuring jailbreaks, but it's an attacker's metric, not a defender's. Just because a specific prompt manages to bypass a guardrail doesn't mean that prompt is actually useful for training a more robust model. If you're using red-teaming data to fine-tune safety alignment, you don't need every successful attack; you need the ones that actually move the needle on model resilience.

This is where a defender-centric approach changes the game. Instead of asking "Did this break the model?", the question becomes "Does including this attack in my training set actually improve downstream safety?"

The A-MESS framework tackles this by using AttackSHAP—a Shapley-based scoring system. For those unfamiliar, Shapley values are typically used in game theory to determine how much each player contributes to a total outcome. In this context, it treats individual jailbreak attacks as "players" and calculates their marginal utility toward improving the model's safety.

The findings here are pretty revealing:

• ASR vs. Utility: High attack success rates correlate weakly with actual safety utility. A "noisy" jailbreak might have a high ASR but provide zero value for alignment training.
• Subset Optimization: Selecting a compact subset of high-utility attacks is significantly more effective than just dumping every successful attack into the training pipeline.
• Efficiency: AttackSHAP can be estimated accurately without needing an infinite number of utility queries, making it practical for real-world LLM agent development.

Essentially, we need to stop treating jailbreaks as just "failures" to be patched and start treating them as high-value data resources. The goal isn't just to stop one specific prompt, but to identify the underlying vulnerability that the prompt exposed.

If you're building a safety pipeline from scratch, focusing on marginal utility rather than raw success rates will likely lead to a much leaner and more effective red-teaming dataset.