VAmoS Bench: Why This Voice Agent Simulation Benchmark Matters

Most voice agent benchmarks are basically eavesdropping tests. They feed a model some audio, check if the transcription matches, and call it a day. That measures ASR, not agency. VAmoS Bench takes a different route — it drops a voice agent into a full simulated environment where it has to actually do things: book a table, dispute a charge, schedule an appointment — and grades the outcome, not just the words.

The core idea is simulation. Instead of static clips, VAmoS creates scripted user personas that react dynamically to whatever the agent says, within a defined goal structure. Every run is a multi-turn dialogue with state. The agent isn't just parsing audio; it's tracking an evolving conversational context while dealing with the realities of voice — disfluencies, interruptions, background noise, or a user who changes their mind mid-sentence.

What I like most is how it separates failure modes. The benchmark scores along a few distinct axes:

• Task success rate: did the agent actually accomplish the user's goal, judged by the environment state, not by whether the user said "great job"
• Grounding performance: how often the agent asked for clarification vs. confidently acting on a wrong assumption
• Latency efficiency: not just model inference time, but how many turns the agent needs to complete the task without repeating itself
• Robustness to ASR errors: the simulator injects realistic transcription noise, so you can see whether the agent recovers from a misheard constraint or spirals

That last one is the real differentiator. A voice agent that can't detect when a keyword got misheard is going to fail in production no matter how good its LLM backbone is. VAmoS makes that failure measurable and reproducible.

Here's a rough idea of what an evaluation config looks like — a scenario definition paired with a success predicate:

{
  "scenario": "restaurant_reservation",
  "user_persona": "impatient_callers",
  "max_turns": 12,
  "inject_asr_errors": true,
  "success_conditions": [
    "booking_confirmed_for_requested_time",
    "no_incorrect_credit_card_charged",
    "user_constraints_satisfied"
  ]
}

The agent runs until the environment terminates, and the benchmark returns pass/fail per condition. Aggregate that across hundreds of scenarios and you get a score that actually correlates with real-world deployment usefulness — something most existing voice leaderboards can't claim.

Is it perfect? No. Simulated users will never fully capture the chaos of real callers — the long pauses, the emotional spikes, the sudden topic jumps. And scenario design sets an upper bound on what gets tested;