CauterRule v0.3.1 shows that 0% pass rates are often data failures, not model ones

A 0% pass rate on a corpus is a question about your data, not a final verdict on the model. In the v0.3.1 field test for CauterRule—an open-source sidecar that converts agent failures into standing rules via extraction and replay-testing—two corpora that previously failed completely were actually victims of "empty haystacks." After fixing the data, the adapters corpus went from 0/60 to 60/60, and raw/ci moved from 0/110 to a 45-55% pass rate.

Why matcher_gap is different from no_signal

When analyzing inconclusive results during replay, it is critical to distinguish between these two attributions:

• no_signal: The model failed to produce a candidate, or the candidate matched nothing when there was actually something to match. This points to a model or gate failure.
• matcher_gap: The model successfully produced a candidate, but the reference pool contained nothing to compare it against. This is a pool failure.

In the adapters corpus, we saw a 100% matcher_gap. The models were actually extracting rules for PydanticAI, CrewAI, and LangGraph, but because the reference pool lacked framework-specific trajectories, those candidates had nowhere to land. The model wasn't failing; the haystack was just empty.

Solving the empty adapters pool

The v0.3.0 reference pool had zero trajectories for the specific frameworks mentioned above. To fix this, I didn't touch the matcher or the model; I authored adapter-specific reference signatures with matching domains, ensuring they had the correct shape, domain labels, and actual failure text.

The jump in performance across both tested models was immediate:

• adapters (n=60): Moved from 0 pass (100% matcher_gap) in v0.3.0 to 60/60 pass in v0.3.1.
• Precision: 1.000
• Recall: 0.168 (gpt) / 0.175 (llama)

Fixing the four-layer failure in the CI haystack

The raw/ci corpus was a bigger mess, moving from 0/110 to 21/47 (gpt) and 26/47 (llama). This wasn't one bug, but four separate data issues that could each independently cause a total failure.

The first and most glaring error was in the collection script. The predecessor to collect-ci-corpus-v2.py was grabbing the first 2,000 characters of the gh run view --log output. Since CI logs start with boot and checkout boilerplate, the actual failure text—which always sits at the bottom—was missing from 99 out of 110 trajectories. The model was essentially being asked to extract rules from empty air.

For those wanting to test the framework, you can get the CLI and rule packs via:

pip install cauterule

The full technical breakdown, including the 4,742 trajectory-runs across 40 corpora, is available in the v0.3.1 field test report.