Stop assuming a model is "blind" to new attacks just because the
Most people see these numbers and immediately conclude the model’s internal representations are insufficient—that the "brain" of the model simply doesn't recognize the pattern of a new injection.
They are wrong.
I’ve found that the issue isn't the representation; it's a massive calibration problem. Here is a quick deep dive into how you can diagnose this and fix it in about 20 minutes without even touching the base model weights.
The 20-Minute Diagnostic
Before you waste weeks fine-tuning a model or switching to a massive, expensive LLM agent for security, you need to run a simple probe. Instead of looking at the final classification output, pull the penultimate embeddings from the frozen encoder.
If you take those frozen embeddings and fit a simple logistic regression on them, the results change instantly. On the same OOD data where the original head failed, a linear probe achieves an AUC of approximately 0.999.
This tells you everything you need to know:
- High AUC + Low Recall: Your classification head or your threshold is miscalibrated. The information is there, but the "decision maker" is tuned too conservatively.
- Low AUC: The model actually lacks the features to distinguish the attack. You have a genuine representation problem and need a different model.
In the case of Prompt Guard 2, Meta made a specific product choice: they tuned the head for extreme precision to ensure a near-zero false-positive rate. They traded recall for stability. That’s a valid design choice for a product, but it’s a nightmare for someone trying to catch evolving jailbreaks.
A Practical Tutorial for Better Recall
If you find yourself in the "High AUC" scenario, you can implement a custom deployment workflow to boost your security posture.
1. Extract Embeddings: Run your training data (both benign and injection samples) through the encoder and save the penultimate layer outputs.
2. Train a Linear Head: Use a simple logistic regression. Since the head is just a dot product, the inference cost is negligible: sigmoid(x·w + b) >= tau.
3. Calibrate on Real Traffic: This is the critical step. Don't just tune on the attack set. Calibrate your threshold ($\tau$) using the benign traffic distribution you actually expect to see in production.
By following this approach, I managed to hit 99.9% OOD recall with only a 0.7% False Positive Rate (FPR). The base model remains completely untouched, and the inference runs perfectly fine on a standard CPU.
The Reality Check
I want to be clear: this isn't a magic bullet that "solves" prompt injection. A linear head over frozen features is still susceptible to evasion if an adversary knows exactly how you've shifted your operating point or if they introduce a massive distribution shift.
What this does is move your operating point from "useless" to "highly effective" by reclaiming the signal that the original developers intentionally suppressed. It turns a failed deployment into a robust defense layer.
If you want to look at the specific implementation and the seeds used for the data splits, you can check the repo here:
https://github.com/mosafariuk/prompt-guard-2-frozen-head