Using Evalgate to Trigger CI Failures When Prompt Quality Regresses

The primary conclusion is that evalgate serves as a critical regression guard, catching silent quality decay before it reaches the end user by treating quality drops as build failures.

Why are traditional unit tests insufficient for prompts?

Quality regressions in LLM outputs rarely trigger exceptions. When you modify a system prompt, swap a model, or integrate a new tool, the resulting JSON typically still parses correctly, meaning no red flags are raised in a standard test suite. The output simply becomes lower in quality, a fact usually discovered by users rather than through Continuous Integration (CI). Because there is no actual crash to catch, unit tests are the wrong instrument for this problem; the failure mode is a decline in performance, not a technical error.

How does evalgate function as a quality gate?

Evalgate addresses this by treating quality drops as build artifacts. It is a lightweight TypeScript CLI that executes a declarative evaluation suite, calculates scores, and maintains a baseline. During every pull request, the tool re-runs the suite and compares the current results against the base branch. If any specific case regresses beyond a defined tolerance level, the process exits with a non-zero status, effectively killing the CI job. To provide visibility, it then posts the resulting scores as a comment on the PR.

The core philosophy is based on a shift in questioning. While asking "Is this prompt good?" is subjective and nearly impossible for an automated gate to answer, asking "Is this prompt worse than it was on the main branch?" is objective and answerable. Evalgate focuses on the latter. By capturing a baseline once, every subsequent modification is judged as a delta against that baseline rather than against an abstract, unattainable ideal.

What does a configuration suite look like?

Suites are stored as YAML or JSON files within version control, situated alongside the code they protect. For example:

name: my-agent
provider: mock # works with no API key
threshold: 0.9 # mean score required to pass
cases:
 - id: greeting
 input:
 prompt: |
 Reply with the standard greeting.
 exactly: Hi there! How can I help you today?
 expected: "Hi there! How can I help you today?"
 scorers:
 - type: exact-match
 - type: latency
 budgetMs: 500

Which scorers are available for validation?

The tool provides a catalog of scorers designed for real-world model output assertions:

• String-level checks via exact-match, regex, contains, and not-contains.
• Structured output validation using json-schema.
• Meaning-based checks via embedding-similarity using cosine similarity.
• Criteria-based evaluations through rubric and llm-judge.
• Budgetary gates using cost and latency to ensure agents do not become too slow or expensive.

How does the mock provider ensure reproducibility?

A standout feature is the ability to run with zero API keys. Evalgate includes a deterministic mock provider that allows the entire suite to be reproduced offline. For embedding-similarity, it uses the provider's embed() method if available, otherwise falling back to a stable local bag-of-hashed-words embedding. While llm-judge normally calls a provider for a {score, reason} JSON response, the mock provider replaces this with a reproducible word-overlap score. This architecture allows the project's own 67 tests to run without ever touching the network.

What is the operational workflow?

The process is managed through three specific commands:

npx @royalpinto007/evalgate run suite.eval.yaml
npx @royalpinto007/evalgate baseline suite.eval.yaml --out baseline.json
npx @royalpinto007/evalgate compare suite.eval.yaml --base baseline.json --tolerance 0.01

The inclusion of a tolerance setting is vital because model outputs are not perfectly stable. This prevents the gate from becoming flaky while still providing a hard stop when an agent truly degrades. Many evaluation frameworks report quality but fail to gate on it; evalgate ensures that a prompt rewrite that makes an agent measurably dumber will fail the build, with the PR comment highlighting exactly which case dropped and by how much. Integrating this into a GitHub Action or an npm run check hook provides the ideal layer for stopping silent rot.