Weather Model Accuracy: Comparing AIFS, GFS, and Commercial Apps

Most weather apps claim high accuracy but treat their data sources like a black box. To get some transparency, I built a scoreboard that actually audits these forecasts. The logic is simple: the system pulls forecasts from various models—including ECMWF, GFS, ICON, AIFS, and commercial providers like Apple Weather, Foreca, OpenWeather, and Visual Crossing—waits for the actual weather to occur, and then scores the temperature, wind, and rain predictions against real-world observations.

Unexpected Findings in Model Performance

Running this analysis revealed some gaps between what is marketed and what actually performs.

• AIFS Performance: The AIFS model is performing remarkably well. What's weird is that almost no commercial apps explicitly give users access to it or admit to using it in their blends, though I suspect some are quietly integrating it.
• Foreca's Edge: Foreca is scoring surprisingly high when compared to both other commercial apps and the raw global models.
• Regional Variance: The ICON model shows a strange split; it's incredibly accurate around the Mediterranean but drops off significantly in quality everywhere else.

I also added a history page that tracks these models back through their archives—roughly five years for GFS—to see if forecasting is actually improving over time or just plateauing.

Technical Implementation and Architecture

This project is built as a static page and is fully open-source. The way data is handled depends on the source:

1. Public Models: These are scored directly in the user's browser by comparing forecasts against the Open-Meteo archive.
2. Commercial Models: Because these providers have strict terms against redistributing raw forecast data, the scores are sent as small aggregates from my server.

The logic is essentially a real-world AI workflow for data validation: taking a prediction, waiting for the ground truth, and calculating the error margin. For those interested in the underlying logic or wanting to see how the scoring is weighted, the code is available here:

https://github.com/NickLeenders/verisky-scoreboard

This scoreboard actually serves as the backend for a more detailed app that performs these same accuracy checks on a per-location basis. It's a much more reliable way to decide which app to trust for your local forecast than just reading the App Store description. If you're into data science or meteorological deployment, this is a great example of how to build a verification pipeline from scratch.