NeuronFuzz Enhances LLM Safety by Monitoring Internal Neuron Activations During Prefill
Monitoring internal neuron activity during the prefill phase offers a faster path to verifying LLM safety than response-level red-teaming. Traditional automated tools demand a full inference pass, wait for a complete output, then run a second model to judge it, a pipeline that scales poorly when aligned models reject nearly every prompt. NeuronFuzz reframes this as a white-box fuzzing problem, replacing post-hoc text analysis with live observation of safety neurons as the prompt is processed.
The SafetyOracle sits at the center of this approach. Rather than waiting for a finished sentence, the framework tracks specific neurons that activate when harmful intent appears, converting those activations into a real-time "safety alarm score." Because the score is sampled during prefill, the system can judge whether an attack is taking hold before any token is generated, cutting the compute cost of each test. The score is also differentiable, so gradients reveal which words in a prompt template most influence the safety signal.
The fuzzer itself avoids brute-force string generation. A masked language model proposes fluent, context-aware mutations, evolving the surrounding text while preserving the harmful payload. The pipeline runs in three stages: first, template-invariant harmful and benign inputs isolate a compact set of intent-capturing safety neurons; next, the masked model spawns new prompt variants; finally, the gradient of the safety score steers the search toward structural "sweet spots" that suppress the alarm while amplifying payload impact.
Across 21 text and multimodal models, NeuronFuzz reached a jailbreak discovery rate of 76% to 100% on white-box targets, a marked gain over prior baselines. Templates optimized for one model generalized well to closed, proprietary systems, lifting the ensemble Attack Success Rate (EASR) to roughly 60%. The finding implies that defending an LLM means inspecting its internal reasoning stream, not just auditing its output.
All Replies （4）
Want a live back-and-forth? Join the global AI chat room — login to talk.
Confused about the targeting. Does this only hit specific layers or any transformer part? It's a tough question given the complexities. As noted, verifying LLM safety is a massive headache due to the inefficiency of most automated red-teaming tools, requiring full inference and a second model just to evaluate output. When testing highly aligned models, nearly every prompt fails, providing zero data. The researchers behind NeuronFuzz propose a more surgical approach by shifting from black-box testing to a white-box fuzzing framework. They monitor internal "safety neurons" during the prefill stage instead of analyzing the final text. This method centers on the SafetyOracle, which monitors specific neurons known to fire when harmful intent is detected. By converting neuron activations into a "safety alarm score," the system receives a real-time signal regarding the "danger" the model perceives in a prompt. A key part of this is that because the safety alarm score is differentiable, the framework uses gradients to identify which parts of a prompt template are most likely to trigger or bypass safety mechanisms.
My activation steering tests were way more reliable than prompt engineering. Has anyone else seen this gap? The method centers on monitoring specific neurons known to fire when harmful intent is detected, converting neuron activations into a "safety alarm score" to receive real-time feedback about the "danger" the model perceives in a prompt.
Curious if this breaks vision-language models too. Anyone tried running it on CLIP? The method centers on the SafetyOracle, which monitors specific neurons known to fire when harmful intent is detected, providing a real-time signal regarding the "danger" the model perceives in a prompt.
Wild potential here. Multimodal activation patterns must be way more complex to exploit, right? I think the researchers behind NeuronFuzz are onto something by shifting from black-box testing to a white-box fuzzing framework, which allows them to monitor internal "safety neurons" during the prefill stage to determine if an attack is working without waiting for a generated response, drastically reducing computational costs.