Video LLMs are failing a basic logic test

The core issue is something called "Understanding-Refusal Coupling Failure." In plain English, the model's vision system sees the danger, but the safety trigger doesn't fire because the text prompt isn't explicitly "aggressive." This creates a massive loophole where a benign-looking query paired with a harmful video is actually more likely to bypass safety filters than a query that is openly harmful.

The V-DEAL Diagnosis

The V-DEAL framework breaks down this vulnerability across three levels: behavior, understanding, and internal representations. The data from six different Video LLMs reveals a weird contradiction:

• Recognition Accuracy: Over 81% (the models know exactly what they are looking at).
• Attack Success Rate: 48.33% (they still provide the harmful output when the query is benign).

When looking at the hidden states, the internal "refusal tendency" is significantly weaker when triggered by visual understanding compared to textual understanding. Essentially, the model's "safety alarm" is tuned to listen to the text, but it's nearly deaf to the visuals.

Fixing the Gap

Instead of relying on expensive fine-tuning, a prompt injection intervention can bridge this gap. By forcing the model to better couple its visual perception with its refusal mechanism, the attack success rate dropped by an average of 48.24 percentage points.

This suggests that the "safety" of Video LLMs isn't about the model lacking the ability to see danger—it's about a failure in the internal wiring that connects "I see something bad" to "I should not answer this." This is a critical insight for anyone working on an AI workflow involving multimodal inputs, as it proves that sanitizing the text prompt isn't enough if the video content itself is the vector.