NeuronFuzz uses internal neuron activations to break LLM safety

Testing whether an LLM is actually "safe" is a massive headache because most automated red-teaming tools are incredibly inefficient. Right now, if you want to test a prompt for a jailbreak, you have to actually run the full inference, wait for the model to generate a complete response, and then run a second model to check if that response was harmful. This is what researchers call "response-level feedback," and it’s a terrible way to scale. If you're testing a highly aligned model, almost every single test prompt will fail, giving you zero useful data on how close you actually came to triggering a violation. You're just stuck in a loop of "Access Denied" messages.

The researchers behind NeuronFuzz have proposed a much more surgical approach by moving from black-box testing to a white-box fuzzing framework. Instead of looking at the final text output, they look at the internal "safety neurons" during the prefill stage.

How the SafetyOracle works

The core of this method is something they call the SafetyOracle. Instead of waiting for the model to finish its sentence, they monitor the activations of specific neurons that are known to fire when harmful intent is detected.

• Continuous Feedback: By converting these neuron activations into a "safety alarm score," the system gets a real-time signal of how much "danger" the model perceives in a prompt.
• No Generation Required: Because this score is obtained during the prefill phase, you don't have to wait for the model to generate a response to know if your attack is working. This drastically cuts down the computational cost of safety evaluation.
• Gradient-Based Mutation: Since the safety alarm score is differentiable, the framework can actually use gradients to see which specific parts of a prompt template are most likely to trigger (or bypass) the safety mechanism.

The fuzzing process

NeuronFuzz doesn't just throw random strings at the model. It uses a masked language model to perform mutations that are fluent and contextually aware. The goal is to keep the harmful payload intact while evolving the surrounding text to slip past the guardrails.

1. Identification: They use template-invariant harmful and benign inputs to find a compact set of safety neurons that specifically capture intent.
2. Mutation: A masked language model generates new variations of the prompt.
3. Optimization: The system uses the gradient of the safety score to find the exact "sweet spots" in the prompt structure that minimize the safety alarm while maximizing the payload's effectiveness.

The results are honestly pretty staggering. In tests across 21 different text and multimodal models, NeuronFuzz hit a jailbreak discovery rate of 76% to 100% on white-box models. That is a massive jump compared to traditional baselines. Even more interesting is the transferability; the optimized templates they found for one model worked surprisingly well on proprietary, closed-source models too, hitting an ensemble Attack Success Rate (EASR) of around 60%.

It's a deep dive into the idea that if you want to secure an LLM, you can't just watch what it says—you have to watch how it "thinks" internally. This kind of white-box fuzzing is likely going to become a standard part of the AI workflow for anyone serious about red-teaming and LLM agent security.