Can we stop just randomly mixing safety data into LLM

Fine-tuning an LLM for a specific task usually kills its safety guardrails. Everyone knows that adding some safety examples back into the training mix helps, but the current industry standard is basically "throw in a random handful of safety pairs and hope for the best." This is inefficient and leaves massive holes in the model's defenses.

The DataRx approach changes this by treating safety as a "missingness" problem. Instead of guessing which examples are useful, it looks at the gap between how a model naturally responds to a risky prompt and how a gold-standard safety reference responds. The clever part is that it doesn't look at the actual words (tokens), which are too discrete and noisy. Instead, it analyzes the high-dimensional hidden representations. By calculating the distance in the embedding space, DataRx identifies exactly where the model's safety capabilities are lacking and samples the specific data needed to fill those gaps.

The efficiency gain here is actually wild. In tests using Llama3-8B-Instruct across seven different tasks, random sampling of safety data left the model vulnerable, with an average attack success rate of 59.23%. By switching to this missingness-aware sampling—using only 1% of additional safety samples from the BeaverTails dataset—that attack success rate plummeted to 13.70%.

This is a massive win for anyone building a real-world AI workflow where you need high task performance without the model suddenly deciding to give out dangerous advice. It proves that data quality and strategic selection beat brute-force data volume every time.

For those of us into LLM agent deployment, this suggests a more surgical way to handle alignment. Instead of bloating your training set with thousands of redundant "I am an AI assistant" responses, you can perform a deep dive into the model's hidden states to find the specific failure points.

If you're setting up a pipeline for task-specific fine-tuning, here is the general logic for implementing a missingness-aware strategy:

1. Generate Baseline Responses: Run your target model on a set of safety-probing prompts.
2. Extract Hidden States: Capture the activations from the final layers for both the model's response and the reference safety response.
3. Calculate the Gap: Use a distance metric (like cosine similarity or Euclidean distance) in the representation space to quantify the "missing" safety signal.
4. Prioritize Sampling: Select the examples with the largest gaps for your fine-tuning set.

This method also stacks well with synthetic data generation. If you find a gap that existing datasets can't fill, you can synthesize specific safety examples to target that exact weakness. It moves safety from a guessing game to a data-centric engineering problem.