Effective Techniques to Clean Noisy RLHF Preference Data

DeepSeek‑V3 and Llama‑3.1 demonstrate that the quality of a preference pair (chosen vs. rejected) is far more important than sheer dataset size. Training a reward model on noisy examples where the chosen response is only marginally better than the rejected one teaches the model to be mediocre.

Benchmark studies of cleaning pipelines have uncovered systematic flaws in human annotation. Annotators often exhibit “positional bias,” favoring the first or the longer response regardless of logical merit. Because of this, many engineers abandon raw human labels and adopt an LLM‑as‑a‑Judge workflow for data distillation; this shift yields the largest gains across benchmarks.

Cross‑Verification Filter

Every preference pair is evaluated by both GPT‑4o and Claude 3.5 Sonnet. When the human label claims A > B but both models rank B higher or call it a tie, the pair is discarded. The most damaging noise comes from false positives: the chosen answer contains a hallucination that the human annotator missed. A typical verification prompt reads:

Compare the following two responses to the prompt: [PROMPT].
Identify any factual errors or logical fallacies in Response A and Response B.
If Response A is labeled “Chosen” but contains a hallucination that Response B avoids, mark this pair as “Noisy”.

This check isolates such noisy examples.

Handling the “Length Bias” Trap

LLMs tend to prefer longer, more verbose answers. In experiments, Gemini 1.5 Pro was especially prone to rewarding extended text that was mostly filler. To curb this, compute the length ratio of chosen to rejected responses. If the chosen answer exceeds twice the length of the rejected one, flag it for manual review or run a conciseness check. When the extra length does not bring a commensurate quality gain, the pair is removed so the model does not learn to “yap” for higher reward.

Comparing GPT‑4o and Claude 3.5 for Data Cleaning

Performance trade‑offs:

• GPT‑4o processes bulk filtering very quickly and excels at spotting formatting mistakes, but it is lenient toward polite yet empty replies.
• Claude 3.5 Sonnet is far more strict about logical consistency. Using Claude to prune the rejected set produced a noticeable jump in MT‑Bench scores because it catches subtle reasoning flaws that GPT‑4o overlooks.

The “Hard Negative” Strategy

Improving performance isn’t limited to discarding bad data; it also involves upgrading the rejected samples. Swapping random rejections for “hard negatives”—responses that are grammatically correct but logically flawed—forces the reward model to learn a sharper decision boundary.

Key Metrics for Evaluating Your Cleaning Pipeline

• Agreement Rate: The percentage of times your LLM‑judge matches the human label. A rate below 70 % suggests that human labeling guidelines were too vague.
• Margin of Preference: Replace binary A/B choices with a 1‑5 scoring system. Discard pairs where the score difference is only one point, as these provide insufficient gradient signal.
• Token Distribution: Ensure the cleaned dataset does not inadvertently favor one response length; otherwise the model will inherit the length bias you tried to eliminate.