AI Estimation Benchmarks: Why Most Tools Fail

IT projects average a 45% budget overrun according to McKinsey and Oxford. This massive gap is exactly why so many AI estimation vendors claim they can read a spec and predict the cost. But after running a rigorous, pre-registered benchmark on nine public datasets, the results are sobering: no model class actually clears the bar.

The core question was simple: can a statistical engine beat standard baselines and human experts on real-world, logged effort data without data leakage? The answer is a definitive no.

The "Passing" Criteria

To avoid moving the goalposts, the success thresholds were committed to git before the experiments began. To pass, a model needed to hit these marks:

• PRED(25): At least 55% of estimates must be within 25% of the actual value (on 2+ datasets per channel).
• MdAPE: Median absolute percentage error must be ≤ 22% (on 2+ datasets per channel).
• Coverage: Actuals must fall within the nominal-90% interval within a 5-point margin.
• Baselines: Must outperform median-by-category and log-size regression.
• Expert: Must beat the recorded human estimate in aggregate.

The testing spanned two channels: tabular project attributes (including COCOMO81 and SEERA) and natural-language requirements (JOSSE and SiP).

How to Actually Validate LLM Agents for Estimation

Most AI benchmarks are inflated by "peeking." If you're building a real-world AI workflow for project estimation, these four methodological choices are non-negotiable to avoid false positives:

1. Time-Ordered Evaluation: Never use random splits. Randomization lets models "peek" at the future. Use rolling-origin cross-validation or leave-projects-out grouped k-fold to simulate a true cold-start scenario.
2. Adversarial Leakage Audits: Many datasets contain attributes that are only known after the project is done (e.g., "team continuity"). If your model knows how many devs left the project, it's not estimating; it's reading the answer key.
3. Conformalized Intervals: Stop relying on point estimates. Using CQR (Conformal Quantile Regression) allows you to calibrate distribution-free intervals and check empirical coverage.
4. Pre-registered Stop Rules: Decide when you've "failed" or "succeeded" before you start the run.

The benchmark is open source under the MIT license and can be reproduced in a pinned environment. For those doing a deep dive into the data, the full repository is available here:

https://github.com/NaCode-Studios/metis-benchmark