Best practices for cleaning noisy RLHF preference data for LLM fine-tuning

DeepSeek-V3 and Llama-3.1 have basically proven that the quality of the preference pair (Chosen vs. Rejected) matters far more than the sheer volume of the dataset. If you're feeding a reward model noisy data where the "Chosen" response is barely better than the "Rejected" one, you're just teaching your model to be mediocre.

I've spent the last few months benchmarking several cleaning pipelines, and the biggest takeaway is that human-annotated preference data is often riddled with "positional bias"—annotators frequently pick the first response or the longer one regardless of actual logic.

To fix this, I stopped trusting raw human labels and started using an LLM-as-a-Judge setup for data distillation. Here is the workflow that actually moved the needle on my benchmarks:

The "Cross-Verification" Filter
I ran my preference pairs through GPT-4o and Claude 3.5 Sonnet. If the human label said A > B, but both LLMs flagged B as superior or called it a "tie," I tossed the sample. The most dangerous data points are the "false positives" where the chosen answer contains a hallucination that the human missed. Using a prompt like this helps isolate the noise:

Compare the following two responses to the prompt: [PROMPT]. 
Identify any factual errors or logical fallacies in Response A and Response B. 
If Response A is labeled 'Chosen' but contains a hallucination that Response B avoids, mark this pair as 'Noisy'.

Handling the "Length Bias" Trap
LLMs have a pathological obsession with length. In my tests, Gemini 1.5 Pro was particularly prone to preferring longer, more verbose answers even if they were fluff-filled. To scrub this, I calculate the length ratio between the chosen and rejected responses. If the chosen answer is >2x longer than the rejected one, I flag it for manual review or run a "conciseness" check. If the quality gain doesn't justify the length, the pair is discarded to prevent the model from simply learning to "yap" to get a higher reward.

Performance Trade-offs: GPT-4o vs. Claude 3.5 for Cleaning

• GPT-4o: Extremely fast for bulk filtering and great at spotting formatting errors. However, it tends to be too lenient with "polite" but empty answers.
• Claude 3.5 Sonnet: Much more pedantic about logical consistency. When I used Claude to prune the rejected set, the resulting reward model showed a significant jump in accuracy on the MT-Bench leaderboard because it was better at identifying subtle reasoning failures.

The "Hard Negative" Strategy
The most effective way to boost performance isn't just removing bad data, but upgrading the "Rejected" samples. I found that replacing random rejected samples with "Hard Negatives"—responses that are grammatically correct but logically flawed—forced the model to learn a much sharper decision boundary.

Key metrics for your cleaning pipeline:

• Agreement Rate: The % of time your LLM-judge agrees with the human label. If this is below 70%, your human labeling guidelines were likely too vague.
• Margin of Preference: Use a scoring system (1-5) instead of binary (A/B). Throw away any pairs where the score difference is only 1 point; these are "noisy" because they don't provide a strong enough signal for the gradient update.
• Token Distribution: Ensure your cleaned dataset doesn't accidentally skew toward a specific response length, or you'll just bake the length bias back into the model.