LLM Evals: Correctness vs Usefulness vs Safety

Shipping an LLM-powered dev tool is easy; proving it actually works across a 100k-line repo without breaking everything is where the nightmare begins. Most teams fall into the "demo trap"—it looks great in a curated screenshot, but the second a real user hits it with a messy codebase, the performance craters. Why? Because we're still treating dev-tool evals like chatbot evals.

A chatbot hallucinating a date is a minor annoyance. A coding agent hallucinating a method call is a broken build or a production incident (and believe me, your on-call engineer will not be amused). We need to stop obsessing over generic benchmarks and start focusing on whether the artifact survives contact with the compiler.

For anyone building a real-world AI workflow or an LLM agent, you have to measure three distinct axes. If you're only measuring one, you're flying blind.

The Three-Axis Framework

Correctness: This is the binary stuff. Does the patch apply? Does the unit test pass? Does the SQL query actually return the requested rows? This is the easiest to automate because you have ground truth. If you aren't using actual test execution to validate your LLM's output, you aren't doing evals; you're just guessing.

Usefulness: This is the "squishy" part where most teams fail. A piece of code can be technically correct but totally useless. For example, providing a 50-line architectural refactor when the dev just wanted a one-line bug fix is a failure of usefulness. It's the gap between "the model is smart" and "the tool is helpful." (Who actually enjoys reading a 14-step plan for a trivial change?)

Safety: The one everyone ignores until a secret is leaked or a rm -rf happens in a production environment. For dev tools, safety isn't about "avoiding offensive language"—it's about blast radius. Did the agent execute a destructive command? Did it scrape a README that contained a prompt injection that hijacked the session?

Why Dev Tools are Different

Generic LLM advice focuses on "vibes" or LLM-as-a-judge scoring. In the dev world, we have something better: the build pipeline. A diff either applies or it doesn't. A test is either green or red.

The danger is that failure modes in developer tools are sharper. A silent failure—like an agent using a deprecated API that still happens to compile but fails at runtime—is far worse than a chatbot giving a slightly wrong summary.

If you're building a deployment pipeline or a coding assistant, your eval suite needs to move beyond "does this look right" to "does this actually work in the environment." Stop trusting the "Verified" scores on a leaderboard and start building a local test suite that treats LLM outputs as untrusted code.