LLM Eval Strategy

Keeping every finding inside a "model-as-judge" setup is a massive waste of compute and time. Most teams treat the judge's output as the final verdict, but that's just paying a premium for a non-deterministic opinion that you have to re-litigate every single time you run an eval. If a judge flags a recurring issue, you shouldn't just log it; you should "ratchet" that finding down into a deterministic check.

In my experience rolling this out with my team, we realized that relying on a judge for everything creates a slow, expensive loop. The goal should be to promote recurring findings into faster, cheaper tiers of verification.

The Evidence Hierarchy

To do this, you have to categorize your checks by how "corruptible" they are, rather than just by cost:

• Tier 1 (Unforgeable Proof): These are binary facts. Did the code compile? Is the JSON valid? Did it timeout? These are free, instant, and deterministic.
• Tier 2 (Statistical Signal): These compare output against a baseline the agent didn't create, such as embedding similarity to a spec or repetition profiles.
• Tier 3 (Model-as-Judge): This is a subjective opinion. It's a signal, not a verdict, and it should stay offline because it's slow and inconsistent.

The rule is simple: Tier 1 and 2 are your real-time gates. Tier 3 is for discovery.

Implementing the Ratchet

When a judge keeps flagging the same failure—for example, "the summary cites a section that doesn't exist"—you don't leave that check in Tier 3. You ask: what factual check would have caught this?

If you can verify that cited headers exist verbatim in the source, you've just promoted a subjective opinion to a Tier 1 deterministic gate.

type Finding = {
 claim: string; // the judge's recurring complaint
 sourceText: string; // artifact the agent did NOT author
 citedSections: string[];
};

// Tier 1 promotion: the finding is now an unforgeable-proof check.
function citationsExist(f: Finding): { pass: boolean; missing: string[] } {
 const missing = f.citedSections.filter(
 (s) => !f.sourceText.includes(s)
 );
 return { pass: missing.length === 0, missing };
}

// Tier 2 promotion: when membership is too rigid, drop to a
// baseline-relative signal the agent didn't get to author.
function citationGroundedness(
 f: Finding,
 embed: (t: string) => number[],
 cos: (a: number[], b: number[]) => number
): number {
 const src = embed(f.sourceText);
 const scores = f.citedSections.map((s) => cos(embed(s), src));
 return scores.reduce((a, b) => a + b, 0) / (scores.length || 1);
}

If exact matching is too brittle, you drop to Tier 2 (like the citationGroundedness function above) to check similarity against the source. It's still deterministic and essentially free compared to an LLM call.

By pushing 80% of your failures into Tier 1 and 2, you stop treating your eval pipeline like a guessing game and start treating it like a deployment gate. Only the truly subtle 20% should ever reach the judge.