Model Swapping: Why "Vibe Checks" Fail for LLM Agents

Changing a model string and reading three responses isn't a test—it's a gamble. In my experience rolling out agents for my team, the most dangerous failures are the ones that look perfect to the user. An agent can tell a customer, "I've successfully cancelled your subscription," while silently failing to actually trigger the cancel_subscription tool call. The prose is polished, but the business logic is broken.

To stop these regressions from hitting production, I've implemented a strict 20-minute diffing protocol. Instead of guessing if a new frontier model is a "drop-in" replacement, I treat the swap as a breaking change until proven otherwise.

The Baseline Capture

The biggest mistake is swapping the model before recording how the current version behaves. Once you change that string, your baseline is gone. I use a recording proxy to capture the exact tool calls and arguments being sent.

First, spin up the recorder:

npx whatbroke-cli record --out baseline.jsonl

Then, point the SDKs to the local proxy:

OPENAI_BASE_URL=http://127.0.0.1:4141/v1
ANTHROPIC_BASE_URL=http://127.0.0.1:4141

I run every critical scenario three times. Because LLMs are non-deterministic, a single run can be a fluke. If a tool call disappears 1/3 times, it's a "flap"; if it disappears 3/3 times, it's a regression. I focus exclusively on "action" scenarios—API hits and database writes—because pure chat rarely reveals the architectural drift that kills an agent.

The Isolated Swap

I follow one golden rule: Change the model string and nothing else.

If you tweak the system prompt to "optimize" it for the new model at the same time you swap the version, you've introduced two variables. You won't know if the new behavior is due to the model's weights or your prompt edit. If the model requires a prompt change to function, that gets its own separate diff.

Executing the Diff

After recording the same scenarios with the new model using npx whatbroke-cli record --out swapped.jsonl, I run the comparison:

npx whatbroke-cli diff baseline.jsonl swapped.jsonl

I categorize the results into three buckets:

• Breaking Changes: These are the red flags. A tool call that existed in the baseline is now missing, or a run that previously succeeded now errors out.
• Argument Drift: This is the sneakiest failure. The tool is still called, but the arguments have shifted (e.g., a date format changed from YYYY-MM-DD to MM/DD/YYYY). This usually breaks the downstream API.
• Performance Ratios: I flag any latency increase over 1.5x or cost increase over 1.25x.

Integrating into the AI Workflow

To prevent "vibe-based" deployments from creeping back into the team's habit, I've moved this into our CI pipeline. By using the --fail-on breaking flag, the build fails if a model swap drops a tool call.

npx whatbroke-cli diff baseline.jsonl current.jsonl --fail-on breaking

For those of us already using observability tools like Langfuse or LangSmith, you don't necessarily need to record new runs. You can export traces from the previous week and the current week to perform a retrospective diff:

npx whatbroke-cli import last-week-export.json --run baseline
npx whatbroke-cli import this-week-export.json --run swapped
npx whatbroke-cli diff last-week-export.whatbroke.jsonl this-week-export.whatbroke.jsonl

This approach turns a subjective "it feels faster/smarter" conversation into a technical report with verifiable data.

https://github.com/arthi-arumugam-git/whatbroke