Guest essay by Eric Worrall
The UK MET is celebrating that their new £97 million computer can now create slightly better 12 month predictions than tossing a coin.
The Met Office has shown it can predict the weather one year in advance with its new £97 million supercomputer.
Scientists believe they can now forecast with some accuracy the North Atlantic Oscillation (NAO) weather phenomenon in the Atlantic Ocean which largely governs the British winter.
The phenomenon forms because of low-pressure over Iceland and high pressure over the Azores in the Atlantic.
A large pressure difference brings increased westerly winds, cool summers and mild, rainy winters. In contrast when the difference is small there are fewer winds and Britain shivers in a big freeze during the winter months.
It was previously thought that the NAO was a chaotic system which could not be predicted but the Met Office has used a technique called ‘hindcasting’ to check whether their new supercomputer could have predicted past winters.
After looking back at weather data going back to 1981, they discovered that they could largely predict what the winter weather would have done for the past 35 years, a year in advance, with 62 per cent accuracy.
The abstract of the study;
Skilful predictions of the winter North Atlantic Oscillation one year ahead
The winter North Atlantic Oscillation is the primary mode of atmospheric variability in the North Atlantic region and has a profound influence on European and North American winter climate. Until recently, seasonal variability of the North Atlantic Oscillation was thought to be largely driven by chaotic and inherently unpredictable processes. However, latest generation seasonal forecasting systems have demonstrated significant skill in predicting the North Atlantic Oscillation when initialized a month before the onset of winter. Here we extend skilful dynamical model predictions to more than a year ahead. The skill increases greatly with ensemble size due to a spuriously small signal-to-noise ratio in the model, and consequently larger ensembles are projected to further increase the skill in predicting the North Atlantic Oscillation. We identify two sources of skill for second-winter forecasts of the North Atlantic Oscillation: climate variability in the tropical Pacific region and predictable effects of solar forcing on the stratospheric polar vortex strength. We also identify model biases in Arctic sea ice that, if reduced, may further increase skill. Our results open possibilities for a range of new climate services, including for the transport, energy, water management and insurance sectors.
Read more (paywalled): http://www.nature.com/ngeo/journal/vaop/ncurrent/full/ngeo2824.html
It will be fascinating to see whether the new forecast system delivers. A lot of models which can hindcast successfully don’t survive contact with reality.
Imagine building a model for predicting lottery wins. With enough effort your model could be coerced into accurately hindcasting past lottery results. But it is very unlikely your lottery model would be able to predict future draws. Fitting a model to a limited data set is often not the same thing as accurately modelling the physics behind the data.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
The Met Office will not be pleased with that telegraph article. It over eggs the pudding somewhat.
When I think of hind casting, this picture comes to mind. http://www.lifeisajoke.com/pictures388_html.htm
And the famous ‘ We are in drought’ London bus in a downpour . . . .
http://randomlylondon.com/is-there-any-doubt-we-are-in-drought/
Auto – well aware that 62% accuracy should beat [I nearly wrote trump(!)] tossing a fair coin.
Not sure it beats – ‘same as before’ – even in the UK . . . .
To be fair the the UK Met Office, their claim that they can predict the state of the NAO with 62% accuracy is pretty impressive.
However, it will give no clue whatsover as to the detail of the winter’s weather.
Given the nature of the climate here in the UK, predicting with any degree accuracy at more than 5 days is while nigh impossible.
So, for that reason, I don’t get what the Met Office has to gain from this claim. They should stick to their short term forecasts, which are very good.
They are back-justifying their need for a new £97m computer, its that’s simple.
Totally agree!
i guess they needed an upgrade so they could play the latest games in decent framerates? :-p
But the figure 62 isn’t that far from 50, is it – which is chance!!
It’s even closer to 66%, which would be pretty impressive for a year in advance.
Statistical significance is not a simple comparison to coin tossing.
In a system with strong auto-correlation I doubt the 62% is random chance but it needs to assessed with more thought than tossing coins.
But this NOT hindcasting since this the calibration period for the model.
Do the same test for 1920-1945 or 1950-1975 and tell us how well it works out.
So what they really have shown is that having tuned to model to a certain period of data , even then it barely reproduce what is happening to better than random variations.
with 62% accuracy is pretty impressive
Suppose they are 100% wrong… How many years do we have to go out before we can invalidate the 62% accuracy? At least 3 years of Wrong, Wrong, Wrong before you can invalidate a success rate of 62% or better at 90% confidence.
I thought I remembered reading was if you predict tomorrows weather was going to be the same as today’s you ended up being right about 60% of the time. Guessing that might work year over year???
I wonder how that 62% compares to the Old Farmer’s Almanac. There is a post in Wikipedia (from sources not on line)
So now we have to define how accuracy is defined. If you predict the weather from weekly averages over the past 30 years, are you accurate if you are near the mean? What if accuracy is defined as within the posted spread — and the posted spread is +/- one standard deviation. By statistical definition, you would be within the range only 68% of the time. And deviate from the range 32% of the time. (assuming normal distribution, which don’t apply to precipitation and temperature uncertainty.)
So let’s see if the UK supercomputer can do better than the Farmer’s Almanac under the same test of accuracy.
It is a worthwhile objective if they can figure out how to do it. Good on them for giving it a go. To a farmer just knowing generally whether to expect a mild wet winter or a dry cold one would be very useful. It could help determine which varieties of crops to plant or how much feed to buy in.
Ian H, various copies of different Farmer’s Almanacs have been providing that “need” for more than 100 years.
Good on them?? It’s only good if it were cost-effective. Spending $97 million to emulate a coin toss gets no kudos from me. Whom the gods would destroy, they first drive mad.
“which are very good”
What do you base that on?
My own analysis of the next 24 hour web forecast for the nearest MO station to where I live (Albemarle) was that MO forecasts were at best 37% accurate.
Based on hourly forecasts symbols v observations for April 2015..
Otherwise they are wrong about 6 out of 10 times on timing and weather type.
Don’t believe what the MO say about accuracy but check out the accuracy for your own location.
I did exactly that for a year, about my home town here in the south of England. I showed their forecast, then the actual weather. They were completely correct only 20% of the time. The problem is that they use words that are get-outs. They would say something like:
‘Most places will see a dry day, but there is the chance of an odd shower here and there’.
That isn’t a forecast. They can’t go wrong with saying that – so I discounted forecasts like that, and they were added to their ‘incorrect’ tally. Sometimes they would get it incredibly wrong, like:
‘A dry day everywhere’.
…And it would rain, maybe only for a few minutes, but it was still wrong.
Or:
‘A showery day’.
And we never had a single shower. When it was warm and sunny, they couldn’t go wrong, and anyone could have predicted it, even a chimp. All in all, I find the Met Office completely and utterly useless. ‘Maybe’, ‘Possibility of’, ‘Chance of’, ‘In places’, ‘Could be’, etc. are not forecasts. You can’t go wrong by saying those things, even if they’re true.
What does 37% accurate mean?
If you could predict the throw of a die 37% of the time it would be quite impressive,
if you could predict the lottery 37% of the time it would be extremely impressive.
Are you arguing that until weather forecasts can predict exactly where each shower will form, they are useless?
Of course it helps that they keep changing the ‘forecast’ regularly by which time what they said 5 days earlier is long since gone. I borrow from Robin Page who in his book commented that the more sophisticated and technology driven the weather forecast has become the worse it has got. I recall a TMS classic when looking ahead to a day’s play, that the forecast they had now was completely different from the one they had been given an hour earlier.
Bolded but not commented on was the solar forcing on the stratospheric polar vortex. I thought their religion forbade any admission of solar effects?
Statistically 62% accuracy is little better than guesswork
62% is a lot better than guesswork when averaged over a long period.
If you could predict correctly whether a roulette wheel finished on black or white 62% of the time you would be rich. If you guessed you would be broke.
“What does 37% accurate mean?”
It means precisely that the symbols were correct 37% of the time.
“If you could predict the throw of a die 37% of the time it would be quite impressive,
if you could predict the lottery 37% of the time it would be extremely impressive.”
But the throw of a die and the lottery is are random events, the weather isn’t.
16.6% would be chance for a die, the MO do twice as well as that.
If the MO can’t forecast the weather to the hour, then they shouldn’t pretend that they can.
“It means precisely that the symbols were correct 37% of the time.”
Which is quite impressive. The MO use a large number of symbols with subtle differences.
To get it right 37% of the time requires predicting the exact type of weather to a specific hour and getting it exactly right more than 1 in 3 times.
“If the MO can’t forecast the weather to the hour, then they shouldn’t pretend that they can.”
I doubt the MO ever claimed they can predict the weather to the hour with the sort of precision you require.
Their forecasts are a good indication of the likely weather patterns over the course of a day – not an exercise in precognition.
Obviously you never have been a forecaster. Because the forecasts (I’m talking about the 3-5 day forecasts) are issued for large geographic areas, they can’t catch the shower that falls on your house. If you need specific forecasts for your immediate are you need to pay a private forecaster who (for a price) can improve the accuracy of the forecast for your house.
I don’t know about the MET office but the NWS forecasters for over at **least** 40 years (that’s when I became an observer in the Navy) have issued forecast reasonings. They are transparent about their confidence in the forecasts, if the models are in disagreement, agreement, or not handling the situation well. If the MET issues forecast reasonings I’d suggest you check into why they forecast as they do.
As to long range >2 weeks, only a fool forecasts long range or the stock market; of which there are plenty in government.
“Are you arguing that until weather forecasts can predict exactly where each shower will form, they are useless?”
I don’t think I am.
But it is the MO who are saying they can forecast weather to the hour, by producing the forecast.
5 days you say . they struggle with rain, wind strength and direction as close as 24 hours out in anything other than weather arriving from a westerly direction. in times of so called austerity that 90 odd million quid was a complete waste of money.
bit chilly
Utter agreement. The MO has good days – and rather bad ones.
If the weather is from the west – and the jet stream isn’t on walkabout – they’re OK for a couple of days . . . .
If.
Good rule for the London commuter – carry a brolly 24/7/365 [366 this year!].
Auto
As for accuracy, I think the UK Met Office can be congratulated for predicting it will be generally colder in winter than it will be in summer.
As for anything else, historical fact has not been kind to previous Met Office’s predictions. Mother Nature will probably take care to ensure it remains that way.
Note how almost everything to do with climate/weather contains the magic number 97.
I am sure the Met Office will be able to predict the arrival of each of the four seasons one year in advance — with 62% accuracy.
Eugene WR Gallun
Don’t bank on it Eugene. I found the Met Office offshore forecasts more wrong that right. One left me beating to windward going up The Channel in what was supposed to be a force 4. It was actually a force 7.
I’ve never trusted them since.
not a nice experience at all . if only met office employees could experience the reality vs their forecasts that some of us do.
Ah, now, like rapscallion and bit chilly, I would be content if the MO could get tomorrow right with – ohhh – say 97% accuracy.
Do a Vivaldi and get four seasons right – well, maybe, but they’ve had over 150 years practice.
Auto – long a weather fan.
And why not?
These Isles get their fair share of it.
I have predicted (with at least 97% accuracy) for many years, that here in Norway we will have winter sometime between october-april, followed by some kind of spring, then summer (june-august) and autumn next. I can even do this without a computer! (No grants for me, then….).
What happened in the 3% of years where spring did not follow winter?
“The Met Office has shown it can predict the weather one year in advance with its new £97 million supercomputer.”
Aah but can they tell me the weather on my day off next weekend?
Cheers
Roger
http://www.rogerfromnewzealand.wordpress.com
The straight answer is NO.
Where I live in South Wales, ie coastal they cannot “Accurately” predict the weather 24 hours before hand.
They get it generally correct, but there hourly timing is usually off and sometimes they get it completely wrong.
You also notice that they change their “5 days forecast” practically every day leading upt to your target day and sometime even on that day.
Sounds very similar to the NZ Met Office:)
It will be interesting to see what odds if any the bookmakers will give on the predictions happening.
I suspect they won’t take bets on anything other than the traditional snow at Christmas.
bookmakers odds reveal bets placed, ie betting publics confidence in the predictions.
Means nothing except as an index of the beliefs of a superstitious sector of the public.
yes, the odds i got on brexit were a good example of that .the “book” was top loaded with larger bets from the liberal islington set types and yuppies in and around the south east of england that wrongly thought they knew better than the rest of the country, as usual. still laughing now 😉
Ah, in fairness to the MetOffice, they made no such claim. They claimed that with a 62% accuracy they can predict the state of the NAO. Only time will tell if they can.
Of course, NAO can offer some clues to what weather can take hold, but no more than that.
With ‘hindcasting’ to test a model, particularly a model that incorporates data from observations and one that is ‘tuned’ [aka fiddle-factors/parameterisations] it is very difficult to exclude the possibility that the results are biased in favour of ‘predicting’ what actually happened.
With the Met Office’s chaotic processes for model development, it is virtually certain that such bias is unintentionally built in to its models.
It is likely that the “Persistence Model” or “Naive” forecast will produce far better than 62% accuracy in forecasting a year in advance, i.e. the weather will be the same on this date a year from now as it is today.
The Met office using very generous, (loose?) metrics in assessing its own accuracy. For example, a temperature forecast is considered accurate if gets within plus or minus 2 degrees Celsius. Even a 3 hour forecast.
I think a Naive forecast would do quite well.
One for British readers.
From a Met Office press release dated 1 April 2018: “Our first 12 month forecast was a great success. We predicted all the right weather. Just not necessarily in the right order.”
Morecombe & Wise, Greig Piano concerto sketch
decnine
Almost wet myself!!
Plus lots of shedloads!
Auto
http://www.metoffice.gov.uk/images/prob_ensemble/20161001/2cat_20161001_temp2m_months35_global_deter_public.png
From the Met Dec – Feb temp anom,
http://www.metoffice.gov.uk/research/climate/seasonal-to-decadal/gpc-outlooks/ens-mean
http://www.metoffice.gov.uk/images/prob_ensemble/20161001/2cat_20161001_mslp_months35_global_deter_public.png
I could be wrong but it seems to be suggesting an Atlantic driven (mild) winter??
How do they get a 2 meter temperature where they don’t measure it, for example, where most of the red and orange is….
Actually, they had applied for a computer of £100 million but the financing body decided that it should allocate only 97% of that amount.
I think Piers Corbyn has been predicting the polar vortex with similar or better accuracy for several years, with the aid of his laptop which may or may not (delete as appropriate) have cost £97 million. It is interesting that they now believe that the sun does have an influence on the AMO and Polar Vortex having stated in the past, I believe, that the forcing is trivial.
I wonder also if they are including their version of CO2 forcing into predictions as they always have done – I am convinced this alone knocks their predictions out of whack relatively quickly. However notice that it isnt really mentioned – could this be because the percentage went up when it was removed – I wonder
looks like they have been listening to ren regarding ozone levels.
“I think Piers Corbyn has been predicting the polar vortex with similar or better accuracy for several years, with the aid of his laptop which may or may not (delete as appropriate) have cost £97 million.”
If only he would publish a paper demonstrating that success.
62% with 97% consensus?
According to the old wives, cold British winters usually come in fours. Is that sufficient for a 62% prediction? I think it is.
What an incredible waste of money , I have a 100% accurate weather instrument , it’s a bit of wool hanging from a small stand .
All you do is put it outside and somehow it starts working straight away .
If the wool is wet it’s raining .
If the wool is moving its windy .
If the wool is missing it’s a hurricane .
If the wool is warm it’s sunny , I have found it to be always accurate and beats the BOM as a matter of fact I think they might now use a similar system .
You forgot If it’s white on top it’s snowing, and If you can’t see it it’s foggy.
“Fitting a model to a limited data set is often not the same thing as accurately modelling the physics behind the data.”
Too true it isn’t. You can fit a polynomial to any data set but unless you understand the physics of why your polynomial fits you’d have to be insane to try using your polynomial for extrapolations into the future. Even if you do understand the physics – or think you do – these kinds of extrapolations are fraught with disaster until you’ve run and compared against real world data and refined over and over and over until it’s demonstrably accurate and reliable.
No, you would be insane to extrapolate you polynomial out to 2100 AD, but it may be a good guide for this winter.
Of course it was sceptical scientists who showed them that these ocean oscillations in both the Atlantic and Pacific were first order drivers that largely explained global weather without the need to evoke CO2 at all. Their pathetic models only 5yrs ago basically employed their much debunked CO2 control know equation which could only give a one way forecast of ever warming weather,even though they used the same 100M pound computer. Over the past 10yrs they would have better done with a coin toss.
Now with the reluctantly accepted knowledge of ocean oscillation elephants in the room, the forecasts can be done using a 300 GBP surface pro or maybe just an eyeballed guess with the status of these oscillations.
I’m grateful these morons are not designing aircraft or bridges: “ah yes, there is a 62% chance the bridge/plane will stay up”. Good grief.
But people know much much more about aircraft and bridge designing and building than they do about the atmosphere and how it works.
i may be wrong osd, but i think that was part of the point bertief was making.
Oh, of course…That’s why the debate is over.
I have just hacked the first UK Met Office 1 year forecast.
Here it is:
Sunny periods with scattered showers.
Brilliant!
Who are their customers? Who is going to act on their predictions? What will they do differently?
It would seem to be a £97 million toy.
Warm and wet vs cold and dry. 62% chance of predicting correctly a year in advance. So even if this is true what possible use will it be? What specific behaviour should I change based on one prediction or the other. What specific behaviour would power suppliers, local councils or any other authority actually change based on such a prediction?
I have tracked the four day UK forecasts that are printed in the Daily Telegraph. These are very broad such as it will rain in north Scotland but it will be sunny in London. They are worthless. They fluctuate violently from day to day. To think it is my money these guys are wasting.