Flaky LLM gates are destroying our build trust

Our CI/CD pipeline turned into a slot machine last week because we relied on an LLM-as-a-judge for our faithfulness gate. We had a hard rule: any merge request with a mean faithfulness score under 0.80 automatically failed the build. I watched a build hit 0.79 and fail, then I re-ran the exact same job with zero code changes and it passed at 0.82. The third attempt sat at exactly 0.80. The judge was hallucinating different scores for the identical input, and that kind of variance is a nightmare for a dev team trying to maintain a stable workflow.

When your automated quality checks are inconsistent, the team stops respecting them. Instead of investigating actual regressions, everyone just starts "re-running until it turns green." You lose the ability to tell if a failure is a real bug or just a statistical fluke.

I spent a few days digging through our setup to find where the noise was coming from. The culprit was a mix of things: non-zero sampling temperature making borderline cases swing wildly, and using model aliases like "gpt-4o" which allows providers to swap the underlying snapshot without telling you. We also had vague rubrics in our prompts that allowed the model's internal logic to drift, and a massive issue with tie-breaking where tiny amounts of noise decided whether we passed or failed.

To fix this for the team, I had to stop treating a single LLM score as absolute truth. You can't get bit-identical determinism from a hosted model, so you have to build systems that manage the jitter. I moved the judge prompt into our repo so it's treated like actual code, not just some random string in a UI, and I pinned the specific model version to prevent drift.

Most importantly, I changed our evaluation logic. We no longer rely on a single call. We run the judge $k$ times and take the mean. To prevent sub-grid jitter from triggering false alarms, I quantized the scores into coarse increments (0.25). Now, we don't fail a build just because a score sits on the threshold. We only fail if the mean drops below the threshold plus the measured variance. It’s the only way to make the pipeline predictable enough for a real production environment.