LLM Drift Tracking: Why My Alerts Were All Wrong

A 35-task evaluation suite cannot resolve anything finer than approximately 2.86 points per question. When my automated drift tracker flagged four regressions this week across three different labs, the math revealed a hard truth: I wasn't measuring model degradation, I was measuring the granularity of my own instrument.

I maintain a public board that probes 16 LLMs on a frozen suite of 35 tasks daily. The system is designed to be autonomous; if a model's score drops compared to the previous run, it automatically opens a GitHub issue and drafts a post. Between July 21 and 24, the system triggered four times:

23 Jul Gemini 3.5 Flash -11.4 pts
24 Jul Gemini 3.1 Pro -2.9 pts
21 Jul Grok 4.3 -5.7 pts
22 Jul Llama 3.3 70B -2.9 pts

On the surface, this looks like a systemic collapse of model performance. In reality, none of these models actually got worse.

The Reliability Gap

The first failure in the alerting logic was a confusion between intelligence and availability. Every score on my board is paired with a reliability metric—the percentage of probe calls that successfully returned a response.

Looking at the logs for the Gemini 3.5 Flash "crash":

gemini-3.5-flash 22 Jul acc 1.000 reliability 1.000
 23 Jul acc 0.886 reliability 0.914 429: Rate limit reached for model `llama-3.3-70b-versatile`
service tier `on_demand` ... requests per minute (RPM): Limit 30, Used 30

The model didn't lose its capabilities; 34 out of 35 calls were rate-limited. Because the tracker scored a 429 error as 0%, the resulting dip looked like a regression. This is the most dangerous type of false positive in an AI workflow: a provider outage or rate limit masquerading as a cognitive decline. Gemini 3.1 Pro followed the same pattern, bouncing back to 97.1% in the next clean run, higher than its previous baseline.

Signal vs. Noise in Small Eval Sets

The other two alerts were more insidious because the reliability remained at 1.000. The numbers were "real," but the interpretation was wrong:

• Grok 4.3: 0.800 -> 0.743 (-5.7 pts)
• Llama 3.3 70B: 0.800 -> 0.771 (-2.9 pts)

In a 35-task suite, one single question represents 2.86 points. A drop of -2.9 points is exactly one question changing its answer. A drop of -5.7 is exactly two.

Reporting a one-question flip as a "model regression" is reading noise as signal. When your eval suite is this small, you cannot distinguish between a genuine drift in LLM reasoning and the inherent stochasticity of the model.

Building a Robust Drift Detection Workflow

The temptation is to lower the sensitivity—perhaps only alert if a model drops by 10+ points. However, that ignores the reality that real regressions often start small. The solution isn't to mute the tracker, but to decouple detection from publication.

My current deployment uses a "human-in-the-loop" stub. The automation logs the event, but the draft contains a mandatory warning:

Auto-logged when the scheduled probe flagged a run-over-run regression. 
Before this becomes a post, check the run log and the Reliability metric 
— a rate limit or provider outage can look exactly like a regression.

The "Alert Survival Rate" Metric

For anyone building a practical tutorial for internal evals, I suggest tracking one specific metric: the share of alerts that survive manual verification.

This week, my survival rate was 0%. While that looks bad on a dashboard, it is the most valuable data point I have. It proves that my suite is too small to resolve single-task noise and that accuracy metrics are lies if they aren't displayed alongside reliability data.

The hardest part of maintaining a deep dive into model performance isn't detecting the drift—it's ensuring you aren't manufacturing it.

https://egnaro9.github.io/model-drift