Reposted from The Cliff Mass Weather and Forecasting Blog
Last quarter I taught Atmospheric Sciences 101 and as a fun extra-credit activity students had the opportunity to participate in a forecasting competition in which they predicted temperatures and probability of precipitation at Sea-Tac Airport. The National Weather Service forecast is scored as well to provide a comparison to highly trained and experienced forecasts. In addition, we averaged the prediction of all the students, producing what is known as a consensus forecasts.
Now who do you think won? The pros at the Seattle National Weather Service office or the average of the inexperienced, weather newbies in my class? The answer is found below–the consensus of the students was considerably superior to the Weather Service folks (click on image to enlarge).
Students were number two overall and the NWS was in sixth place.
A fluke ? No–it happens this way virtually EVERY YEAR. To illustrate this, here are the results for 2004. In that year, the average of the students was fourth, the NWS experts were in 10th place. You will notice that some individual students sometimes came in first or ahead of the NWS…that could be just random luck due to the brevity of the forecast contest (1-1.5 months).
This phenomenon is often called the Wisdom of Crowds and has been the subject of a number of journal articles. So why might an average of the students be better than a NWS forecaster? Some possibilities include:
1. The average forecast of a group will tend to damp out forecast extremes, which produce very bad scores when they are wrong.
2. Students look at many different sources of information, using weather information in different ways and viewing many different forecasts (e.g., from various private sector groups). Forecasts derived from an average of many different sources tends to be more skillful on average.
3. Some of them might have took at look at superior forecasts, say form weather.com or accuweather.
I can think of other possibilities…perhaps you can too.
This wisdom of crowds finding is closely relate to why we make ensemble forecasts, running models many times, each slightly differently. The average of these many forecasts is on average the best forecast to use.
So next time you need a forecast, trying averaging the guesses of your friends or classmates. Does this idea apply to elections? Now that is a subject I think I want to avoid.
Same for my area.
And the local channels’ forecasters usually beat both hands down.
When I moved to Seattle about 45 years ago I noticed how bad the weather forecasts were, compared to NY City. Perhaps they’ve got a little better recently, but not much. I think they are locked into some approach that “ought” to work but doesn’t. Maybe it’s a highly “scientific” approach that is preferred because it is not feasible for outsiders to use it—IOW, because the official forecasters have an “exclusive.” This would make psychological sense.
Roger – Seattle is more difficult to forecast for than NYC. To the uninformed it’s a conspiracy.
The idea would apply to elections if voters could choose between eight parties and twenty-odd candidates, without continual brainwashing by the MSM.
As it is, voters get two choices imposed on them by ‘experts’ (rich folks with ulterior motives) who choose solutions that suit them, not the majority.
I think that this bit:
“This wisdom of crowds finding is closely relate to why we make ensemble forecasts, running models many times, each slightly differently. The average of these many forecasts is on average the best forecast to use.”
is entirely misguided if applied to climatic forecasts. For weather-type models, which can get checked with real world data in a day or two so that they can be tuned/trained to local conditions, this may work thru an evening out of the spread of possible solutions. (Weather models that throw out really wrong forecasts are corrected until they can produce close-to-correct forecasts. When model ensembles are used in weather forecasting, they represent the combined opinions and views on weather of their makers/progrsammers and thus may have some Wisdom of Crowds advantage.
However, for climate forecasts, “Many models” producing “many forecasts” and/or the same models running many forecasts “each slightly differently” will give us nothing but the averages of chaotic output as the models pass the point in future time in which Sensitivity to Initial Conditions throws the model into a chaotic mode. In the chaotic mode, models are not even really processing the climate, but their own internal chaotic functions.
Kip – There are constraints on chaotic activity (if we don’t heat the mid troposphere too much).
You’re a denier and you’re worried about cascades?
In John Brunner’s SF novel “Shockwave Rider”, he had “Delphi Pools” which tried to forecast using crowds. Apparently everyone knows what’s going on, without every ONE knowing…