SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation

SimuVerity is a new benchmark that tests agent systems' ability to generate Simulink models meeting engineering standards. Traditional benchmarks only check if models compile, run or look like a reference, not if they work for real engineering needs. SimuVerity covers 101 text-to-executable Simulink model tasks across ten domains like control and power systems. Each task has profiles defining the engineering requirements and four native simulation scenarios. The evaluator first checks if the model is delivered, executable, and meets engineering specs, then rates qualified models on six dimensions: accuracy, output quality, mechanical correctness, control/causal integrity, robustness in operating conditions, and dynamic response. Six agent systems were tested using SimuVerity. The best achieved only 42.86 overall. This shows that even if models look similar, they often fail to meet real engineering demands due to issues in both implementation and post-qualification multidimensional requirements. Some high-scoring models still had severe visual layout problems. SimuVerity helps diagnose failures and assess an agent's true engineering capability in executable Simulink model generation tasks.
What are the main goals of the SimuVerity benchmark?

SimuVerity aims to evaluate how well AI agent systems generate Simulink models that are not just executable but also meet real-world engineering standards. It focuses on diagnosing failures in model creation beyond just compilation or similarity to a reference. The benchmark uses a hierarchical evaluation framework that first checks if the generated model is deliverable, can run natively, and satisfies key engineering requirements. Then, it rates qualified models across six dimensions:

1. Accuracy: How closely the model's outputs match real-world measurements or specifications.
2. Output quality: The fidelity and quality of generated outputs to engineering standards.
3. Mechanistic fidelity: How well the model captures the underlying mechanisms and physics.
4. Control and causal integrity: The model's ability to correctly model causal relationships and promote sufficient control over the simulated system.
5. Operating-domain robustness: Performance and reliability in the environment or domain for which the model is designed.
6. Dynamic response: The model's accuracy in capturing system dynamics and response to various input scenarios.

These dimensions ensure that Simulink models generated by agents are assessed in a multidimensional way, reflecting the complexity of engineering-grade requirements. SimuVerity helps uncover bottlenecks in producing qualified models and meeting performance demands even after initial qualification.

What are some of the key results from testing agent systems with SimuVerity?

When six agent systems were tested using SimuVerity, the benchmark revealed that even the best performer achieved only a 42.86 overall score. This demonstrates that structural similarity alone is a poor indicator of engineering performance, as many generated models passed basic criteria but failed to meet the multidimensional requirements of real-world applications. Several capability bottlenecks were identified:

1. Qualification failures: Difficulty in producing models that meet basic engineering specifications, meaning many models couldn't advance to the detailed evaluation stage.
2. Post-qualification shortfalls: Even qualified models struggled with accuracy, robustness, and dynamic response, highlighting issues in generatinghigh-quality implementations.
3. Layout and presentation: Instances where high-scoring models exhibited severe visual layout disorder, making them hard to use despite good technical metrics.

SimuVerity thus provides a clear measure of current agent capabilities in generating engineering-grade Simulink models and offers a framework for identifying areas needing improvement.