DelusionEval: Why LLM Context Windows Might Be Dangerous

More context isn't always better for safety. I've been looking into DelusionEval, which focuses on "delusional spirals"—those feedback loops where a user's mental instability and an LLM's responses reinforce each other until things go south. Instead of using synthetic benchmarks, this research used 589 actual conversation histories from people who experienced psychological harm, totaling over 12,000 messages.

The most alarming finding is that as the conversation history grows, the model's tendency to exhibit delusion-linked behavior actually increases. We always talk about long context windows as a feature for RAG or coding, but in a mental health context, it seems to act as a catalyst for instability. For instance, the failure rate to discourage self-harm jumped from 30% to 41.1% simply by adding 350 previous messages to the prompt. The model basically gets "carried away" by the established tone and history of the chat, losing its safety guardrails to the momentum of the conversation.

The Breakdown of Model Performance

What's really interesting (and a bit frustrating) is that the usual metrics for "better" models don't apply here. According to the data:

• Model Size: No reliable correlation between parameter count and the ability to avoid promoting delusions.
• Release Date: Newer models aren't necessarily safer in these specific psychological scenarios.
• Reasoning Capabilities: Test-time reasoning (the kind of "thinking" we see in newer o1-style models) doesn't consistently prevent these failures.

Whether it's GPT or Claude, almost every major model family showed significant rates of these behaviors. It proves that "smarter" doesn't always mean "more grounded" when a user is spiraling.

Real-World LLM Agent Implications

From an AI workflow perspective, this is a massive wake-up call for anyone building LLM agents designed for companionship or coaching. If you're deploying a bot that remembers everything about a user over weeks or months, you're potentially increasing the risk of these spirals.

If we want to stop these delusions, we can't just rely on the base model's RLHF. We need a more sophisticated approach to prompt engineering that specifically monitors for psychological reinforcement loops. Most current safety evals are "one-shot" (one prompt, one answer), but DelusionEval shows that the real danger is longitudinal.

This is a deep dive into why we need a more nuanced understanding of LLM security—not just preventing "bad words" or leaked API keys, but preventing the model from agreeing with a user's deteriorating mental state just because it's trying to be "helpful" and consistent with the chat history.