Visual inputs are bypassing LLM safety filters in ways that

I've been fascinated by why Vision-Language Models (VLMs) are so much harder to secure than pure LLMs. You can spend weeks fine-tuning a text model to refuse harmful instructions, but as soon as you feed that same concept into a VLM via a cleverly disguised image, the whole safety layer seems to crumble. A recent research paper finally looked under the hood to see if these models actually share "safety neurons" across modalities, or if the visual side of the brain is operating on a completely different logic.

The researchers performed a deep dive into 10 different VLMs to see how visual inputs perturb safety pathways at the actual neuron level. They didn't just look at the outputs; they used a two-stage detection pipeline with iterative ablation to see which specific neurons were responsible for saying "no."

The findings are honestly a bit of a wake-up call for anyone working on AI alignment or red-teaming. Here is the breakdown of what they discovered:

• Text Safety is Highly Concentrated: In these models, text-based refusal is driven by a tiny, incredibly specific group of neurons. We are talking about roughly 88 neurons—which is less than 0.01% of the total—that, when ablated, significantly reduce the model's ability to refuse harmful text. It's a very localized, efficient pathway.
• Visual Safety is Massive and Diffuse: This is where it gets messy. Unlike text safety, which concentrates in about 5 subspace directions, visual safety is high-dimensional. To disrupt visual safety, you need to mess with at least 50 different directions. It is scattered across the architecture in a way that makes it incredibly hard to pin down.
• The "Dominant Pathway" Problem: The study found that text safety neurons are the dominant refusal pathway. If you ablate those specific text neurons, the model stops refusing almost everything, even when the input is visual. This suggests that the model is essentially relying on its text-trained "moral compass" to handle visual inputs, rather than having a dedicated, robust visual safety mechanism.

This explains exactly why we are seeing such a massive "visual safety gap." Current alignment techniques are mostly focused on text. When we train a model to be safe, we are mostly teaching those few dozen "safety neurons" how to react to words. But because visual safety is so diffuse and spread out across the weights, those text-based interventions don't translate well to images.

If you want to look at the technical implementation or the benchmarks they built (ViSafe-Detect and ViSafe-Eval), you can find the full project details here:

https://jiaxuan-li.github.io/vlm-safety-neuron/

It feels like we've been trying to fix a complex, multi-dimensional problem using a one-dimensional tool. Until we figure out how to target that high-dimensional visual safety space, VLMs are going to remain much more vulnerable to multimodal jailbreaks than their text-only cousins.