Local AI
1.17°C
mean error · AIFS on an RTX 4070 Ti SUPER
- rmse
- 1.40
- bias
- -0.84
- worst
- 3.69
- wins
- 18
Open log · updates itself
Weather scoreboard
The weights of ECMWF’s forecast model are public, and they weigh under a gigabyte. So I set it running on my desktop graphics card, and every day I record two things before anything happens: what that model predicts at my house and what the official forecast predicts, which runs on a supercomputer.
When time passes, what actually occurred gets written down. Predictions are never edited. In a few months this table will say who gets it right more often, without anyone having to be taken at their word.
Next hour awaiting confirmation: 20/8, 20:00, 13.4 °C predicted by the model at home.
Official
1.19°C
mean error · ECMWF operational IFS
- rmse
- 1.50
- bias
- -0.81
- worst
- 3.30
- wins
- 20
38 hours verified2 waiting on the clocksince 2026-08-16Santiago, Chile · temperature at 2 m
The three curves
- Local AI
- Official
- What happened
The reality line stops where time has not arrived yet.
The error by how far ahead you look
Each pair compares the mean error at that horizon. The further the model looks, the more it shows.
The full log
The actual column gets filled in later, with the ERA5 reanalysis; until that arrives the recent analysis is used, marked with an asterisk. The two forecast columns are never edited.
Analysis of 2026-08-19 06:00:00 UTC6/8 verified
22.3s on NVIDIA GeForce RTX 4070 Ti SUPER · 5.77 GB of memory· grid -33.583, -70.720
| Hour (Chile) | Local AI | Official | Actual | AI error | Official error |
|---|---|---|---|---|---|
| 19/8, 08:00 | 6.41 | 6.50 | 8.90 | 2.49 | 2.40 |
| 19/8, 14:00 | 16.81 | 18.30 | 17.70 | 0.89 | 0.60 |
| 19/8, 20:00 | 11.65 | 12.10 | 12.20 | 0.55 | 0.10 |
| 20/8, 02:00 | 7.82 | 8.70 | 10.50 | 2.68 | 1.80 |
| 20/8, 08:00 | 6.71 | 7.80 | 10.40 | 3.69 | 2.60 |
| 20/8, 14:00 | 18.56 | 19.20 | 16.90 | 1.66 | 2.30 |
| 20/8, 20:00 | 13.40 | 13.40 | — | — | |
| 21/8, 02:00 | 9.40 | 8.40 | — | — |
Analysis of 2026-08-18 06:00:00 UTC8/8 verified
21.3s on NVIDIA GeForce RTX 4070 Ti SUPER · 5.77 GB of memory· grid -33.583, -70.720
| Hour (Chile) | Local AI | Official | Actual | AI error | Official error |
|---|---|---|---|---|---|
| 18/8, 08:00 | 6.21 | 3.80 | 7.10 | 0.89 | 3.30 |
| 18/8, 14:00 | 17.95 | 17.60 | 17.00 | 0.95 | 0.60 |
| 18/8, 20:00 | 12.64 | 11.80 | 13.40 | 0.76 | 1.60 |
| 19/8, 02:00 | 8.30 | 7.80 | 9.60 | 1.30 | 1.80 |
| 19/8, 08:00 | 6.81 | 6.00 | 8.90 | 2.09 | 2.90 |
| 19/8, 14:00 | 16.40 | 17.20 | 17.70 | 1.30 | 0.50 |
| 19/8, 20:00 | 11.34 | 12.40 | 12.20 | 0.86 | 0.20 |
| 20/8, 02:00 | 7.32 | 9.70 | 10.50 | 3.18 | 0.80 |
Analysis of 2026-08-17 06:00:00 UTC8/8 verified
19.1s on NVIDIA GeForce RTX 4070 Ti SUPER · 5.77 GB of memory· grid -33.583, -70.720
| Hour (Chile) | Local AI | Official | Actual | AI error | Official error |
|---|---|---|---|---|---|
| 17/8, 08:00 | 7.25 | 8.30 | 8.10 | 0.85 | 0.20 |
| 17/8, 14:00 | 13.76 | 13.50 | 13.50 | 0.26 | 0.00 |
| 17/8, 20:00 | 10.76 | 10.30 | 11.60 | 0.84 | 1.30 |
| 18/8, 02:00 | 7.51 | 6.40 | 8.10 | 0.59 | 1.70 |
| 18/8, 08:00 | 6.39 | 3.80 | 7.10 | 0.71 | 3.30 |
| 18/8, 14:00 | 17.94 | 17.60 | 17.00 | 0.94 | 0.60 |
| 18/8, 20:00 | 12.58 | 11.80 | 13.40 | 0.82 | 1.60 |
| 19/8, 02:00 | 8.09 | 7.80 | 9.60 | 1.51 | 1.80 |
Analysis of 2026-08-16 06:00:00 UTC8/8 verified
19.4s on NVIDIA GeForce RTX 4070 Ti SUPER · 5.77 GB of memory· grid -33.583, -70.720
| Hour (Chile) | Local AI | Official | Actual | AI error | Official error |
|---|---|---|---|---|---|
| 16/8, 08:00 | 4.67 | 5.10 | 4.90 | 0.23 | 0.20 |
| 16/8, 14:00 | 14.79 | 15.20 | 14.00 | 0.79 | 1.20 |
| 16/8, 20:00 | 10.21 | 9.80 | 10.80 | 0.59 | 1.00 |
| 17/8, 02:00 | 7.55 | 8.40 | 8.90 | 1.35 | 0.50 |
| 17/8, 08:00 | 6.59 | 7.90 | 8.10 | 1.51 | 0.20 |
| 17/8, 14:00 | 13.62 | 12.70 | 13.50 | 0.12 | 0.80 |
| 17/8, 20:00 | 10.47 | 10.40 | 11.60 | 1.13 | 1.20 |
| 18/8, 02:00 | 6.94 | 5.70 | 8.10 | 1.16 | 2.40 |
Analysis of 2026-08-16 00:00:00 UTC8/8 verified
108.5s on NVIDIA GeForce RTX 4070 Ti SUPER · 5.77 GB of memory· grid -33.583, -70.720
| Hour (Chile) | Local AI | Official | Actual | AI error | Official error |
|---|---|---|---|---|---|
| 16/8, 02:00 | 5.77 | 6.00 | 6.60 | 0.83 | 0.60 |
| 16/8, 08:00 | 5.15 | 5.10 | 4.90 | 0.25 | 0.20 |
| 16/8, 14:00 | 15.17 | 15.20 | 14.00 | 1.17 | 1.20 |
| 16/8, 20:00 | 10.43 | 9.80 | 10.80 | 0.37 | 1.00 |
| 17/8, 02:00 | 7.59 | 8.40 | 8.90 | 1.31 | 0.50 |
| 17/8, 08:00 | 6.56 | 7.90 | 8.10 | 1.54 | 0.20 |
| 17/8, 14:00 | 12.88 | 12.70 | 13.50 | 0.62 | 0.80 |
| 17/8, 20:00 | 10.10 | 10.40 | 11.60 | 1.50 | 1.20 |
Method and data
- Local model
- ECMWF AIFS Single 2.0, public weights (CC-BY-4.0), running on an RTX 4070 Ti SUPER with the wrapper that swaps flash-attn for SDPA.
- Official reference
- ECMWF operational IFS via Open-Meteo, requested with
cell_selection=nearestandelevation=nanto get the grid point without elevation correction. Without those parameters the value arrives adjusted by a 90 m elevation model, which introduces 0.60 °C of bias in Santiago. - Ground truth
- ERA5 reanalysis. While ERA5 is unavailable the recent analysis is used, flagged as provisional and replaced as soon as the definitive one arrives.
- Point
- The nearest point of the reduced Gaussian grid to Santiago (−33.4489, −70.6693), about 15 km from the centre. No interpolation.
- Metrics
- MAE is the mean absolute error. RMSE squares before averaging, so it punishes large errors more: if it is well above the MAE, the model fails rarely but badly. Bias carries a sign and says whether it runs warm or cold.
- Known limitation
- The IFS updates four times a day and the scoreboard queries the current forecast, which may start from a more recent analysis than AIFS does. That favours the official one. Pinning both runs to the same initial analysis is still pending.
Download the full log as CSV· 40 rows · regenerated on every update
How it is built, step by step, in the article about the model.The code that keeps this log lives onGitHub.