Confidence Intervals in LLM Evals: The Clustering Trap

My recent eval pass rates were lying to me because I was treating clustered data as independent samples. If you're calculating binomial confidence intervals (Wald, Wilson, or Clopper-Pearson) based on a simple pass/fail rate across a dataset, you're likely underestimating your variance and overstating your confidence.

The problem is that LLM benchmarks often have "clustered" examples—multiple prompts derived from the same document, or several variations of the same logic puzzle. These aren't independent trials. If the model fails on one version of a specific complex logic pattern, it's highly likely to fail on all others in that cluster.

When examples are dependent, the effective sample size is much smaller than the raw count. This means the confidence interval shrinks artificially, giving a false sense of precision.

The Diagnosis

I noticed a massive discrepancy between my "stable" eval results on a curated set and the actual performance drop-off when I introduced a small set of truly novel tasks. The math was technically correct, but the assumption of independence was wrong.

To fix this in a real-world AI workflow, you have to account for the intra-cluster correlation. Instead of a standard binomial test, I've started moving toward:

1. Bootstrapping by cluster: Resample entire groups of related prompts rather than individual rows.
2. Mixed-effects models: Treating the cluster as a random effect to get a more honest standard error.

If you're running a deep dive into model regressions, stop trusting a 95% CI that ignores how your test cases are grouped. You're not seeing the true margin of error.