Fake APIs miss critical agent failures—real staging clusters expose the truth
When an agent interacts with mocked APIs or simulated databases, the evaluation process doesn’t test the agent itself—it only verifies how well the mocks were constructed. Dor Cohen, monday.com’s AI Engineering Director, emphasized this point during a September 15 discussion, warning that evaluation results become meaningless if the testing environment fails to replicate production conditions.
Surface-level checks miss hidden failures
Unlike traditional model testing, assessing an agent is far more complex because it combines the model, its tools, and external dependencies. Agents also behave differently each time they run, making it impossible to rely solely on whether the final response sounds correct. Instead, the entire execution path must be examined.
Cohen pointed to a specific case where an agent was assigned to retrieve 600 items. It executed the required tools, generated a seemingly accurate response, and passed a basic evaluation. However, closer inspection revealed it had only processed 500 items. A standard evaluation focusing only on the output would have missed this critical shortfall, proving that trajectory-based testing is essential.
Mocks create false confidence with three key flaws
While mocks are convenient for speed and simplicity, they introduce three major issues that undermine agent testing:
- Version mismatch: Mocks quickly fall out of sync with production because updates require manual adjustments.
- Overly clean data: Synthetic datasets lack the irregularities and edge cases found in real-world databases.
- State inconsistency: Mocks either fabricate responses or forget previous writes, failing to replicate how agents interact with persistent data.
Staging environments address these problems by using real data at production scale, ensuring agent actions have tangible effects. The trade-off is that staging requires more setup and isn’t as quick to deploy as mocks.
mirrord bridges the gap between local testing and production
To streamline pre-production testing without manual cluster configuration, monday.com uses mirrord. This tool makes a local process or CI job appear as if it’s already running inside a Kubernetes cluster, granting access to real environment variables, secrets, and filesystem operations. To prevent conflicts between evaluation runs, they implement database branching and queue splitting, isolating writes and avoiding unintended overlaps.
Relying on simulated environments to determine agent readiness introduces unnecessary risk. For accurate results, real staging with production-like conditions is the only reliable approach.
All Replies （3）
Want a live back-and-forth? Join the global AI chat room — login to talk.
Frustrated by this too. I wasted a whole sprint on mocks only for the real API to return a 429. Next time, I’ll run the eval in a staging environment that mirrors production and track the entire trajectory instead of grading only the final answer. Maybe using Terraform?
I'm curious if this scales. Does this approach crash under high load with something like k6? Based on what I understand, if your agent talks to fake APIs and mocked databases, you aren't testing the agent—you're testing your ability to write mocks. A session on September 15 with Dor Cohen, AI Engineering Director at monday.com, offers a sharp conclusion: eval results lack meaning unless the environment mirrors production. Testing an agent is far more difficult than testing a vanilla model because an agent bundles the model, its tools, and the dependencies behind those tools. Agents are non-deterministic and may take different paths on each run, so you cannot simply check if the final response "sounds" correct. You must track the entire trajectory. Dor highlighted a clear failure case. Their agent was tasked with retrieving 600 items. The agent called the tools, produced a confident-looking response, and appeared correct on the surface. The catch? It only processed 500 items. A standard eval focused solely on the final output would have passed it, but a trajectory-based eval catches the incomplete job. Mocks are basically fairy tales. Mocks get used because they're fast and lightweight, but they're fundamentally flawed for agent testing for three reasons: - Drift: Mocks never stay in sync with production because updates require manual effort. - Sterile Data: Synthetic data in a mock is too clean, missing the messy edge cases present in real databases. - State Amnesia: Mocks can't truly represent state. If an agent writes data and reads it back, the mock either fakes the result or forgets it happened. Staging environments fix this by running the agent against a subset of production data and infrastructure, which is why you should use them for agent evaluation instead of mocks. You can simulate high-load scenarios by increasing the number of concurrent users in a staging environment, ensuring the agent interacts with a more realistic setup than mocks can provide.
Finally some sanity. I spent weeks debugging a mock that worked but failed on the real v2.1 API. Anyone tried using LocalStack? If your agent talks to fake APIs and mocked databases, you aren't testing the agent—you're testing your ability to write mocks. A session on September 15 with Dor Cohen, AI Engineering Director at monday.com, offers a sharp conclusion: eval results lack meaning unless the environment mirrors production.