LLM judges are too flaky for single-run evals

If you run an LLM-as-judge evaluation twice on the exact same input, you will eventually hit a scenario where the first run passes and the second fails. This randomness makes it nearly impossible to build a reliable CI/CD pipeline because a "green" build might just be a lucky coin flip. I've been looking into how muteval handles this in mutation testing, and it basically treats the LLM judge as a noisy signal rather than a source of truth.

Use majority voting to kill noise

The only way to get a stable verdict is to stop trusting a single response. Instead of one call, evaluate each mutant N times. A mutant is only marked as "killed" if a strict majority of the runs agree. If it's a tie, it survives. This stops a single hallucination or random token shift from flipping your entire test result.

Surface the flakiness as a separate metric

Don't just hide the variance in the average. You need to explicitly flag any mutant that flipped between "caught" and "missed" across runs. If 10% of your mutants are flaky, your judge is unstable. This is actually more useful than the final score because it tells you exactly where your prompt is failing to be deterministic.

Stop reporting fake precision

Reporting a single percentage (e.g., "85% mutation score") with a noisy judge is dishonest. Because the sample size of mutants is usually finite and the judge is probabilistic, you should use a Wilson 95% confidence interval. If your interval is huge, it's the system telling you that you don't have enough data to trust the number.

Short-circuit with deterministic checks

To save on API costs and reduce noise, run rule-based checks before the LLM judge. If a simple regex or a deterministic assertion kills the mutant, you don't need to pay for a GPT-4o call. This minimizes the amount of time your eval spends in the "probabilistic zone."

The trade-offs and failures

This doesn't fix a bad prompt. If your judge is systematically biased or consistently wrong, majority voting just gives you a "stable wrong answer." It solves noise, not inaccuracy.

Also, the cost scales linearly with the number of runs. If you have 100 mutants and 5 test cases, running each 3 times for a majority vote means 1,500 LLM calls. That adds up quickly in terms of both latency and budget.

For those implementing this, here is the logic for the voting mechanism:

def get_stable_verdict(results):
    # results is a list of booleans: [True, False, True]
    # True = Mutant Killed, False = Mutant Survived
    killed_count = sum(results)
    total_runs = len(results)
    
    if killed_count > (total_runs / 2):
        return "KILLED"
    else:
        return "SURVIVED"

# Example of flakiness detection
def check_flakiness(results):
    return len(set(results)) > 1

If you're using a tool like muteval, you can find the implementation here:

https://github.com/AshwinUgale/muteval