Combatting AI Spear Phishing: Moving Beyond Static Rule-Sets

The rise of AI-driven spear phishing has rendered traditional security checklists nearly obsolete. We are seeing a shift where attackers use LLMs to mirror a target's tone and cadence perfectly, making "spot the typo" a useless strategy. AegisAI recently secured $36M to scale their approach of using AI agents for anomaly detection, and while their internal architecture is proprietary, the core challenge they've solved is one many of us are currently struggling with: detecting subtle social engineering without triggering a flood of false positives.

If you are attempting to build a similar filtering layer into your own AI workflow—perhaps to scrub incoming leads or high-value emails—you’ve likely encountered the "Precision Wall." When you prompt a model with generic instructions like "detect anomalies" or "look for social engineering," the LLM typically defaults to two extremes. It either becomes too permissive, letting everything through, or too rigid, flagging every email that lacks formal corporate syntax.

The problem is that "anomaly" is too abstract for a model to quantify without a reference point. To move toward an AegisAI-style detection system, you have to stop treating the LLM as a binary classifier and start treating it as a behavioral analyst.

One effective way to refine this is by implementing a "Contrastive Analysis" loop. Instead of asking the model if a single email is suspicious, you provide the agent with a small corpus of known-good communications from the target's actual history. You then prompt the model to identify "linguistic drift."

For example, if you are using GPT-4o or Claude 3.5 Sonnet, avoid the phrase "Is this a phishing email?" Instead, try a prompt structure that forces the model to score specific dimensions of the text:

1. Urgency Delta: Does the level of urgency deviate from the established baseline of this sender?
2. Request Deviation: Is the sender asking for an action (e.g., clicking a link, changing a wire transfer) that contradicts their historical behavior?
3. Syntactic Shift: Are there subtle changes in punctuation or greeting habits that suggest a different author?

When I tried implementing this, I found that asking for a "Confidence Score" from 1-10 often led to "hallucinated certainty," where the model would give a 9/10 for a false positive. A better approach is to require the model to provide a "Reasoning Trace" before the final verdict. By forcing the agent to cite specific phrases that feel "off" compared to the baseline, you can significantly reduce the false positive rate.

If you're running this via a Python script, I recommend using a Pydantic schema to enforce a structured output. This prevents the model from giving you a conversational "I think this might be phishing" and instead forces a JSON response with a risk_score (integer) and a justification (string).

The goal isn't to create a new checklist—which is exactly what the industry is trying to move away from—but to create a dynamic baseline. The "subtle anomalies" AegisAI refers to are not fixed patterns; they are deviations from a specific individual's digital fingerprint. If you can feed your agent enough context about what "normal" looks like for a specific user, the anomalies become glaringly obvious, even when the grammar is perfect.