CauterRule 0.3.0 fixes the recall bug that made good rules look like noise

I've been digging into CauterRule to turn agent failures into standing rules, and I hit a wall where my golden recall was stuck at 0.087. The initial instinct is usually to blame the matcher or the model, but the actual issue was how the replay metric was calculating the denominator.

If you're using this to extract lessons from trajectories, you've probably noticed that a rule might prevent three real failures—which is objectively useful—but then get a recall score of 0.015. That happens because the system was checking that specific rule against the entire corpus, including domains it was never meant to handle. It's basically asking if a Git-specific rule can prevent a Docker failure. Of course it can't, but the old metric penalized the rule for that.

Why the global denominator kills your metrics

The problem is a flawed assumption that every failure in the reference pool is a fair test case for every candidate rule. When you're dealing with domain-specific guidance, a global pool makes every useful rule look like a failure.

In the v0.2.0 logic, recall was calculated as:
recall = prevented / total_failures

If a rule prevents 3 failures in a pool of 200, you get 0.015. Since you can't set a threshold that low without letting in actual noise, these rules get discarded. The model isn't "failing" to find the rule; the metric is just lying about the rule's value.

How domain-scoped replay actually works

In v0.3.0, the reference pool is filtered to the source trajectory's domain before the replay happens. This means a Git rule is only judged against other Git failures.

Here is the logic change in the codebase:

# scope the comparison set to the candidate's own domain
references = [t for t in reference_pool if t.domain == candidate.source_domain]
prevented = sum(1 for t in references if rule_matches(candidate, t) and not t.success)
recall = prevented / len([t for t in references if not t.success])

By narrowing the slice, the math changes completely. A rule preventing 3 Git failures is now measured against ~27 Git-related failures rather than the 200+ global failures. That moves the score from 0.015 to 0.11.

Implementation details and the threshold shift

If you are updating to v0.3.0 via pip install cauterule, be aware that the pass threshold has been dropped from 0.8 to 0.5. This isn't a "tweak" to make numbers look better; it's a calibration correction. Once the recall is calculated honestly against the domain scope, the old 0.8 threshold was revealed to be inflated.

The impact of this change was tested across 4,768 trajectory-runs and 40 different corpora. In the field tests, the reference pool actually grew from 230 to 444, but because the candidates are now judged against their own domains (e.g., Python rules against Python failures, Docker against Docker), the signal-to-noise ratio finally stabilized.

If you're seeing your rules get rejected despite them clearly solving the problem in the source trajectory, check if your domain tags are correctly assigned. If the domains are mismatched, the scoped replay won't find the references it needs, and you'll be right back at that 0.087 recall floor.