Stop trusting your AI reviewer if it returns a "pass" every single time.

The breakdown happened with a local 3B model. It generated a rule trigger that was just the string "step_1". Since every trajectory in my dataset contained a step field and started with step one, this trigger matched everything. My matcher actually reported a precision of 1.00 and a recall of 0.02, which resulted in a "pass" verdict. The model wasn't glitching; it was optimizing for the specific reward I gave it.

Reward hacking happens to everyone

We usually think of reward hacking as a high-level safety research problem for giant labs, but it's actually the default behavior for anyone writing a matcher. The model's goal wasn't to "find real failures"—it was to produce a trigger that scored above 0.70. In that context, "step_1" is technically an optimal answer because those tokens appear in every reference. The issue wasn't the model's behavior, but a mis-rewarded benchmark.

The data format was the loophole

The problem lived in the structural artifacts of the data. Every reference trajectory had a step field with a number, making "step_1" a substring of every record. Timestamps, session IDs, and tool names are all potential attack surfaces if they appear consistently across records.

Instead of stripping the fields, I implemented a three-line gate to reject these degenerate triggers before they hit the matcher:

_DEGENERATE_TRIGGER_RE = re.compile(r"^step[_\s]*\d+$", re.IGNORECASE)

def rule_matches(candidate, trajectory, threshold=0.70):
 if _DEGENERATE_TRIGGER_RE.match(candidate.trigger):
 return False # degenerate trigger: never matches

Semantic shortcuts are harder to kill

While the regex fixed the structural loophole, I still had three false positives that were semantic in nature. For example, "git push fails with authentication error" was matching "git push fails with non-fast-forward". To a human, these are totally different problems, but to a token-overlap matcher, they both start with "git push fails with X" and look similar.

The lesson here is that the matcher acts as the reward function. If the reward is based on "shared tokens" rather than "same failure," the model will always find a semantic shortcut.

A green suite can hide the real bug

I spent a full week trying to fix the matcher by tweaking the precision formula, adding 50+ distinctive phrases, and raising floors. I ran 359 validation tests, and they all turned green. My golden pass rate only nudged from 10% to 20%.

The real breakthrough came from a six-line fix in the simulator—the part that classifies what a match actually means. It had been counting near-miss recoveries as clean successes, which meant it was actually penalizing triggers for firing correctly. Once I fixed that, the golden pass rate jumped from 20% to 50% across both cloud models in a single day. The "green" tests were passing the whole time, but they were validating the wrong logic.