Stop lying to yourself with mocks and use a staging cluster for agent evals

If your agent is calling fake APIs and mocked databases, you aren't actually testing your agent—you're just testing how well you can write a mock. I caught a session from September 15 with Dor Cohen, the AI Engineering Director at monday.com, and the takeaway is pretty blunt: your eval outcomes are useless unless the environment actually mirrors production.

Why grading the final answer is a trap

Testing an agent is a nightmare compared to testing a vanilla model because an agent is a whole package of the model, its tools, and the dependencies behind those tools. Since agents are non-deterministic and can wander down different paths every time you hit run, you can't just check if the final response "sounds" right. You have to track the entire trajectory.

Dor pointed out a perfect example of this failure. They had an agent tasked with retrieving 600 items. The agent called the tools, formatted a confident-looking response, and looked totally correct on the surface. The problem? It only actually processed 500 items. A standard eval that only looks at the final output would have given this a gold star, but a trajectory-based eval catches the fact that the job wasn't actually finished.

Mocks are basically fairy tales

We all use mocks because they're fast and lightweight, but they're fundamentally flawed for agent testing for three reasons:

• Drift: Your mocks are never actually in sync with production because you have to update them by hand.
• Sterile Data: Synthetic data in a mock is way too clean. It doesn't have the messy edge cases that actually live in your database.
• State Amnesia: Mocks can't truly represent state. If an agent writes data and then tries to read it back, the mock either fakes the result or forgets it happened entirely.

Staging environments fix this because they run real data at production scale and the agent's actions actually land. The only real downside is that staging isn't "lightweight" or easy to wire up for a quick test.

Solving the connectivity mess with mirrord

To get around the headache of wiring up pre-prod environments, monday.com uses mirrord. It basically tricks a local process or a CI job into thinking it's already deployed inside a Kubernetes cluster. It pulls in real env vars, secrets, and remote filesystem reads/writes. To stop different eval sessions from overwriting each other and causing a total meltdown, they use DB branching and queue splitting to isolate the writes.

If you're still relying on a "simulated" environment to tell you your agent is production-ready, you're just gambling with your deployment.