Years ago, I set up a simple website that screen-scraped the BBC's weather predictions, and compared them against the day's weather report to calculate a very crude and basic accuracy.
For the UK towns it monitors, a dumb prediction of "tomorrow's weather will be the same as today's" gives a 34% accuracy - which only falls to about 25% when predicting the weather for next week! Luckily, the proper weather forecasters do a bit better than this :)
Excuse the basic site, I set this up over 17 years ago, and with minimal tweaks it has been left to its own devices since then.
The stats also compare the BBC accuracy over the last year vs all time, and it seems that they are getting better - I wonder if new AI techniques will really make a big leap in predictions or whether they are just more incremental improvements.
During World War II, [Nobel laureate, Ken] Arrow was assigned to a team of statisticians to produce long-range weather forecasts. After a time, Arrow and his team determined that their forecasts were not much better than pulling predictions out of a hat. They wrote their superiors, asking to be relieved of the duty. They received the following reply, and I quote "The Commanding General is well aware that the forecasts are no good. However, he needs them for planning purposes."
1. Subjectively this years weather predictions have been way off, compared to the years before. I heard several theories on that: (a) the year was extraordinary (less cars, less flights) and (b) predictions were worse because data from plane based weather radar was missing.
-> Does anybody know if my subjective feeling is based in reality? And if true, what are the reasons?
2. Again subjectively, but I feel like most of my weather based decisions are "do I leave now or do I wait for the rain to pass". That question is answered pretty well by looking at the weather radar maps myself. I feel like an statistical/ML/AI approach that combines what was the weather yesterday and what is the weather in the surrounding cities should fair pretty well.
1. The data I scrape could be enough to check this theory out. Currently the site calculates a 'all time' accuracy and 'last year' accuracy, it would be fairly simple to also add an accuracy for 1 year back, 2 years back, etc. When I have time, I'll give that a try.
2. I once knew of a website that did just this - it displayed the radar image for the village that the creator lived in, and used some really simple linear motion estimation to predict rain in the next hour. I believe it had pretty good accuracy, but unfortunately I can't find that site any more, sorry.
> (b) predictions were worse because data from plane based weather radar was missing
Is plane based radar even used for forecasting? I cant think of what advantage it would have over satellite and ground based radar, with the possible exception of data gathered midocean (where land radar doesn't exist, but there also arent many people).
Sort-of; it's really the TAMDAR [1] system taking measurements of humidity and temperature that are readily assimilated into the global forecast models run by all of the major weather forecast centers (NOAA, UKMO, ECMWF, etc). These observations play a non-trivial role in improving the assimilated initial conditions and boosting forecast quality. Recent work [e.g. 2] has demonstrated that the degradation in availability of aircraft-based observations during the pandemic likely did produce a real, statistically significant decrease in average forecast skill during the afflicted time periods.
Very cool! Nothing wrong with crude; something crude that exists is better than something polished that does not exist!
I am curious about your implementation of 'accuracy':
> How do I measure 'accuracy'?
> Very simply! I take the BBC's weather icons and compare them, using a bit of leeway. So if the prediction is 'Partly Cloudly', then 'Sunny Intervals' is also considered equivalent. Likewise, 'Light Showers', 'Light Rain' and 'Drizzle' are all considered close enough to be an accurate forecast.
> E.g. as I write this, the table below shows that the weather forecast for Cambridge one day ahead was 53% accurate. In other words, the BBC's guess about tomorrow's weather in Cambridge was right roughly half of the time.
So no partial credit, then? Check my understanding: I think that you're simply matching the title text of the icon. If it's a match (or in a small group of synonyms) that's a point, if it's not, you score zero for that prediction. Yesterday, the forecast for today was "Partly cloudy", today, the actual weather was "Sunny" - it gets no credit.
The parent article neural network is, apparently, scoring itself on matching the radar results pixel by pixel and color by color, which is pretty neat. I think it's particularly interesting if it's essentially general-purpose, taking in one collection of input pictures and outputting another, or whether they also gave it information on high and low pressure zones, prevailing winds, bodies of water and elevated land masses, and so on.
Regardless, what I personally want to know (and what I think most people want to know) from the weather forecast is whether it's going to be suitable for a particular activity. Obviously, the hard part is that the activities may vary for each consultation. If it's predicted to be partly cloudy and mild, and was actually sunny and hot, I'd be pleasantly surprised if I had scheduled a day at the beach, but disappointed if I was sweating while working on some landscaping. Farmers want it wet in the summer for growth and dry in the fall for harvesting, sailors want to know the minimum wind, painters want to know the maximum wind; everyone has different goals day by day.
Have you ever graphed the accuracy over time for the years you've been doing it? It would be interesting to see if there's a trend in forecasting improvement.
For the UK towns it monitors, a dumb prediction of "tomorrow's weather will be the same as today's" gives a 34% accuracy - which only falls to about 25% when predicting the weather for next week! Luckily, the proper weather forecasters do a bit better than this :)
https://weather.slimyhorror.com/
Excuse the basic site, I set this up over 17 years ago, and with minimal tweaks it has been left to its own devices since then.
The stats also compare the BBC accuracy over the last year vs all time, and it seems that they are getting better - I wonder if new AI techniques will really make a big leap in predictions or whether they are just more incremental improvements.