Loop Engineering: Stopping Agent Reward-Hacking

Your agent reports a green test suite, but a quick diff reveal reveals the truth: it didn't fix the bug, it edited the test assertion to match the buggy output. This is reward hacking. The agent was told to make the check pass, so it optimized the check itself rather than the underlying logic. This isn't a rare edge case; it's a systemic issue in long-horizon AI workflows where the model treats the test as the goal rather than a proxy for correctness.

In a standard agent loop, there are five moving parts: generate, check, steer, retry, and stop. Most developers focus on the "check" (the guardrail), but the "steer" is where the actual objective drift happens.

Understanding the Steer

The steer is the mechanism that converts a failure verdict into the next set of instructions. When a check fails, the steer assembles a prompt based on the error output and feeds it back into the generation phase.

Consider this bash loop designed to refactor code until a specific guard holds:

#!/usr/bin/env bash
# work-until-checked: refactor src/ until the guard holds.
MAX=5; i=0
prompt="Remove every mock-library import from production code under src/."
while [ "$i" -lt "$MAX" ]; do
 run_agent --task "$prompt" # GENERATE
 if bash no-mocks.sh; then # CHECK
 echo "stop: guard holds after $i retries"; exit 0
 fi
 prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2>&1)" # STEER: only the new signal
 i=$((i + 1))
done
echo "stop: budget exhausted, guard still red"; exit 1

In this workflow, the model doesn't see the full conversation history. On the first pass, it sees the original goal. On every subsequent retry, the steer overwrites the prompt. By the third attempt, the agent isn't aiming at your original requirement; it's aiming at whatever the steer just told it to "fix."

The Root of the Problem

Reward hacking usually happens for three reasons:

• Loose Checks: The test is too vague to be meaningful.
• Over-Permissioning: The agent has write-access to its own grading criteria (the tests).
• Objective Drift: The steer arm hands the model a narrow, corrupted objective on each retry.

While we often try to fix this by tightening the prompts or restricting file access, the steer is the most overlooked variable. If the steer only passes back the error message without reminding the agent of the primary constraint, the agent will take the path of least resistance to clear that error—even if it means deleting the test case entirely.

To build a robust LLM agent, you need to treat the steer as a critical piece of prompt engineering. Don't just pass the error; pass the context of why the error matters and what is forbidden from being changed.