evalgate: Fail CI on Prompt Regression

Quality regressions never throw exceptions. Swap a model, reword a system prompt, add a tool, and the JSON still parses — nothing goes red. The output just gets quietly worse, and you find out from a user, not from CI. Unit tests are the wrong tool here because there is no crash to catch. The failure mode is a drop in quality, not an error.

evalgate treats that drop like a build artifact. It's a small TypeScript CLI that runs a declarative eval suite, scores it, stores a baseline, and on every pull request re-runs the suite and compares the delta against the base branch. When any case regresses past a tolerance, the process exits non-zero and kills the CI job — then it posts the scores as a PR comment.

The design is smart about what question it asks. "Is this prompt good?" is subjective and unwinnable for an automated gate. "Is this prompt worse than it was on main?" is objective and answerable. evalgate answers the second one. You capture a baseline once, and every subsequent change is judged as a delta against that baseline, not against an abstract ideal.

A suite is a YAML or JSON file living in version control next to the code it guards:

name: my-agent
provider: mock # works with no API key
threshold: 0.9 # mean score required to pass
cases:
 - id: greeting
 input:
 prompt: |
 Reply with the standard greeting.
 exactly: Hi there! How can I help you today?
 expected: "Hi there! How can I help you today?"
 scorers:
 - type: exact-match
 - type: latency
 budgetMs: 500

The scorer catalog covers what you'd actually assert about model output:

• exact-match, regex, contains, not-contains for string-level checks
• json-schema for structured output validation
• embedding-similarity for "close enough in meaning" using cosine similarity
• llm-judge and rubric for softer, criteria-based evaluation
• latency and cost for budget gates, so slower or pricier agents fail too

The most interesting part is that it runs with zero API keys. evalgate ships a deterministic mock provider that makes the whole suite reproducible offline. embedding-similarity uses the provider's embed() if available and falls back to a stable local bag-of-hashed-words embedding otherwise. llm-judge calls a real provider and parses a {score, reason} JSON response, but with the mock provider it computes a reproducible word-overlap score instead. The project's own 67 tests never touch the network.

The workflow is three commands:

npx @royalpinto007/evalgate run suite.eval.yaml
npx @royalpinto007/evalgate baseline suite.eval.yaml --out baseline.json
npx @royalpinto007/evalgate compare suite.eval.yaml --base baseline.json --tolerance 0.01

The tolerance matters because model output isn't perfectly stable. You don't want a flaky gate, but you do want a hard stop when an agent actually degrades. That's a practical tradeoff many eval frameworks skip — they report quality but never gate on it. Here a prompt rewrite that makes an agent measurably dumber fails the build, and the PR comment shows exactly which case dropped and by how much.

I'd probably wire this into a GitHub Action or a simple npm run check hook before trusting it as the only signal, but as a regression guard it's the right layer — the one that catches silent rot before your users do.