Climate models outperformed by random walks

First, a bit of a primer. Wikipedia describes a random walk is a mathematical formalisation of a trajectory that consists of taking successive random steps. For example, the path traced by a molecule as it travels in a liquid or a gas, the search path of a foraging animal, the price of a fluctuating stock and the financial status of a gambler can all be modeled as random walks. The term random walk was first introduced by Karl Pearson in 1905.

Example of eight random walks in one dimension starting at 0. The plot shows the current position on the line (vertical axis) versus the time steps (horizontal axis). Source: Wikipedia

Computer models utterly fail to predict climate changes in regions

From the Financial Post: A 2011 study in the Journal of Forecasting took the same data set and compared model predictions against a “random walk” alternative, consisting simply of using the last period’s value in each location as the forecast for the next period’s value in that location.

The test measures the sum of errors relative to the random walk. A perfect model gets a score of zero, meaning it made no errors. A model that does no better than a random walk gets a score of 1. A model receiving a score above 1 did worse than uninformed guesses. Simple statistical forecast models that have no climatology or physics in them typically got scores between 0.8 and 1, indicating slight improvements on the random walk, though in some cases their scores went as high as 1.8.

The climate models, by contrast, got scores ranging from 2.4 to 3.7, indicating a total failure to provide valid forecast information at the regional level, even on long time scales. The authors commented: “This implies that the current [climate] models are ill-suited to localized decadal predictions, even though they are used as inputs for policymaking.”……

More here: http://opinion.financialpost.com/2012/06/13/junk-science-week-climate-models-fail-reality-test/

h/t to WUWT reader Crispin in Waterloo

Previously, WUWT covered this issue of random walks here:

Is Global Temperature a Random Walk?

UPDATE: The paper (thanks to reader MT) Fildes, R. and N. Kourentzes, 2011: Validation and forecasting accuracy in models of climate change. International Journal of Forecasting. doi 10.1016/j.ijforecast.2011.03.008

and is available as a PDF here

Advertisements

  Subscribe  
newest oldest most voted
Notify of
old construction worker

Climate Models: If a frog had wings,……….

One of the reasons that climate models fail is that they include CO2 which in fact lags temperature rather than being useful in a forecasting model for temperature, rain, wind etc. The so-called climate scientists basically have no understanding of heat transfer and reaction kinetics. As Willis has been showing they need to look at heat from the sun, and clouds in their various forms. They also need to look at the chemical reactions at the top of the atmosphere such as ozone formation.

Heh — the magenta random walk produced a pretty decent hockey stick…

One has to admire McKittrick’s clear and direct style.
I like this long-term precipitation record – Central England & Wales, 1766-now:
http://climexp.knmi.nl/data/pHadEWP_monthly_qc_mean1_1766:2011.png
There is some multidecadal oscillation, but nothing exceptional whether it was LIA or modern WP.

LevelGaze

This surprises me – and probably most people here – not one bit.
As long as the input assumptions are flawed, all that the climatologists will manage with larger and more expensive supercomputers (as demanded by the UK MET) will be to compound their errors.

Helen Ap-Rhisiart

Surely a diagram on a flat surface is in two dimensions?

gopal panicker

the models are screwed up…we have been saying that for a long time…there are too many factors…many poorly understood…co2 is already absorbing all the infrared it can…adding more will not make any difference…these climatologist guys dont seem to understand basic physics

Roger Longstaff

For some time I have been unable to understand how the UK Met Office models can predict climatic conditions decades in the future, when they are admitted to be of “low skill” (by Richard Betts) in predicting the weather / climate over periods only weeks or months in the future. I asked Richard for references to the modelling procedures and he kindly provided several on various threads at Bishop Hill. This raised more questions and I finally emailed the Met Office with the following:
“I would be grateful if you could let me know if you think that it is reasonable practice to use numerical models for multi-decadal forecasts that:
A. Use low pass filters “chosen to preserve only the decadal and longer components of the variability” ( http://www.geosci-model-dev.net/4/543/2011/gmd-4-543-2011.pdf (quote from page 21), and,
B. Accommodate errors that cause instabilities “by making periodic corrections, rather
than by fixing the underlying routines (http://www.cs.toronto.edu/~sme/papers/2008/Easterbrook-Johns-2008.pdf (quote from page 7).”
The Met Office kindly replied with the following answers:
“(A) The models themselves do not use low pass filters. Indeed they simulate weather on timescales of minutes. Low pass filters are used to analyse the model output in order to focus on timescales of interest.
(B) The mathematical equations describing the evolution of the atmosphere and oceans cannot be solved analytically. Instead they must be solved numerically using computers. This process frequently involves a compromise between accuracy and stability. During model development much research is undertaken to find the numerical schemes that provide the best accuracy whilst minimising instabilities. At the Met Office the same numerical schemes are employed for weather and climate predictions, and the performance of the forecasts are continuously assessed across a range of timescales from days to decades.”
I am still struggling to find an answer to my original question, and perhaps my questions to the Met Office were the wrong ones. Is there anybody here who understands how numerical models can be “low skill” over short timescales but could possibly be accurate enough to justify massive intervention over multi-decadal timescales?

Maus

LevelGaze: “This surprises me – and probably most people here – not one bit.”
Yeh no. I’m not simply surprised but mortified. It’s one thing to hold that simply gluing together words ‘climate’ and ‘science’ doesn’t mean there’s much of either in ‘climate science’. It’s quite another to state that the “consensus science” is so stupidly distant from a random walk, on the wrong side of things, that we’re beyond Jonestown and that we’ve gone beyond Ludicrous Speed.
Each of these models, which the article refreshingly notes, are a scientific theory about the climate. And they are when empirically tested make errors more than twice that achieved by a monkey throwing darts. Under any notion of science every one of those models is a miraculously falsified theory.
And of course it serves to note that the average stock broker does slightly worse than an actual monkey throwing darts at a dartboard — for those that remember the MonkeyDex. Slightly worse. And it’s a felony for stock brokers to make too much of their own prognostications without a disclaimer to the effect that they are not Delphic Oracles, Palm Readers, or merely reliable. And that’s just for ‘slightly worse’.
So while I’m unsurprised that the models are terrible, I’m utterly aghast that they are so grotesquely biased as to make the static on an empty television channel the acme of rigorous statistical predictions in climate.

D. J. Hawkins

,b>Helen Ap-Rhisiart says:
June 14, 2012 at 12:43 am
Surely a diagram on a flat surface is in two dimensions?

Yes, but only ONE of the dimensions is length. The other is time. Think of a bead on a string. At time t=0 the bead is at zero, at time t=1 the bead is at -1 (say), at t=2 the bead is at +5, and so on. The graphs shows the position of the bead on the string as a function of time.

Helen

Oops. Brain now switched on.

SteveE

Could someone share the title of the paper this was based on? I’ve looked through the 2011 paper in the forecasting journal but couldn’t find any likely titles. Perhaps I just missed it.
Cheers!

Doesn’t surprise me in the least.
All the climate models do is project the modellers belief in the net forcing of various factors, primarily CO2, forward into the future. That the models do worse than random chance is proof that some or all of the believed forcings are wrong, and specifically the believed CO2 forcing is wrong.
No other conclusion is possible, except perhaps serious programming incompetence by the modellers.

son of mulder

In a nutshell a random walk has a finite chance of being correct but if you start with a wrong model it will always be wrong.

This is the best ever article on climate modelling (although Ross McKitrick’s article just published comes a close second).
Please read it carefully – it’s worth it –
http://climateaudit.org/2011/05/15/willis-on-giss-model-e/

Letmethink

@ Roger Longstaff
“Is there anybody here who understands how numerical models can be “low skill” over short timescales but could possibly be accurate enough to justify massive intervention over multi-decadal timescales?”
No, because this is not remotely feasible. The climate is a chaotic system so a tiny variation in an initial condition will result in entirely unpredictable (or random) outcomes.

Philip Richens

Roger Longstaff says @ June 14, 2012 at 1:03 am
Hi Roger,
Pretty certain that what RB has in mind is that (A) short term random noise (white noise) prevents models from making accurate forecasts over weeks and months and (B) the white noise is averaged out over climate time scales (decades) and hence the models should end up providing reasonable average behaviour over the longer time scales.
The reason that (B) is wrong is simply that the noise in the system does not average out over the decadal time scales. Over these time scales, the pink noise mentioned in the “Is Global Temperature a Random Walk?” link begins to dominate. Models are not currently able to simulate this behaviour, which means that their use in decadal projections is unlikely to be useful globally, let alone regionally.

DavidA

It’s not too surprising; a random simulation can be right some of the time, but a model rigged to be wrong will consistently be wrong.

DEEBEE

GCMs are like communism, they require global venue to succeed as their supporters say; but fail miserably at a local level as reality states.

DirkH

This is great news! With this breakthrough in modeling, we will be able to improve climate forecasting while at the same time reduce the amount of computational performance needed, making for more sustainable modeling. Simply replace the current climate models with random walk models, do an ensemble run and average over it.
I can’t wait to see the headlines:
“Scientists baffled – New Climate Model predicts no change in temperature ever!”

The UK. Met. Office have yet to learn that when you have dug yourself into a hole you stop digging.
Their reply to Roger Longstaff is typical of QUANGOs which the Met Office has become. At least when it was officially under RAF control it did apologize for errors it made.
For non UK readers a QUANGO is a Quasi Non Governmental Organization that is formed to advise the government and in doing so get paid money. The Met Office has dreams of operating the worlds biggest climate computer so needs to scare the government into paying more money. It can’t be bothered with members of the public, who actually provide the money through taxation, asking what it considers stupid questions.
Keep trying Roger.

People here waste too much time in armchair speculation on why climate models can’t work “in theory”. The key issue is validation. Why should I believe what this model tells me? What is the evidence for it’s efficacy? If I have to evaluate the merits of a complex surgical procedure I want to look at the study results. I don’t care if the surgeon is the world’s greatest or that medical science is amazingly good, or if there is a consensus on something related to it, or any other non sequiturs of that type.

Random walk? What random walk? The yellow line and the black line OBVIOUSLY prove my **ULTIMATE THEORY!**
If you people just can’t see that, well, I hope Schroedinger’s Cat bites you!
– MJM

Shevva

Back to discuss about random walks after these ad’s….
‘the financial status of a gambler ‘, I spend 4 pounds a week on the lottery and always lose (using this years figures as I ain’t won a penny) and my friend lost the £120,000 given to him when his farther passsed away plus the £40,000 he stole from work, so I promise you gamblers don’t generally have random walks just declines and even when they do win these are generally smoothed out by, yep, the loses.
Welcome back…..

Man Bearpig

Yes,, this is exactly how they are supposed to work.
If you have two climate models, one says ‘rains and floods will increase’ and another that uses the same data says ‘droughts and hot weather will increase’ then depending on the weather at the time you just refer to the model that got it correct and blame it on climate change. I thought you all knew the science behind AGW ? /sarc

DR_UK

The paper is in the International Journal of Forecasting (not the Journal of Forecasting)
Fildes, Robert and Nikolaos Kourentzes (2011) “Validation and Forecasting Accuracy in Models of Climate Change International Journal of Forecasting 27 968-995.
http://www.sciencedirect.com/science/journal/01692070/27/4
Ross McKitrick’s fully referenced article is here (the journal reference is incorrect there too and in the FP article, but the footnote is correct…)
http://www.rossmckitrick.com/uploads/4/8/0/8/4808045/fp_june12_models_i.pdf

Roger Longstaff

Thanks Philip & Letmethink.
My problem is that any filtering (between calculation steps), or “re-setting” of variables to preserve stability, inevitably leads to loss of information on the system.
Filtering is used to increase signal to noise ratio when there is “a priori” knowledge of the signal (for example in a radio receiver that uses a narrowband filter tuned to a specific frequency). With climate models there is only an assumption that there will be a GHG signal. Furthermore, any low pass filtering will remove information generated by the model itself, as signals must be sampled at twice their highest frequency component (Nyquist theory) in order to preserve information.
If one were to equate accurate information to Shannon entropy, it seems inevitable that climate models will deviate from reality exponentially with respect to time, as a consequence of the logarithmic nature of information theory.
Is there a flaw in my argument?

Don K

Roger Longstaff says:
June 14, 2012 at 1:03 am
I am still struggling to find an answer to my original question, and perhaps my questions to the Met Office were the wrong ones. Is there anybody here who understands how numerical models can be “low skill” over short timescales but could possibly be accurate enough to justify massive intervention over multi-decadal timescales?
=======
Roger
In my part of the world there is a dramatic difference between Summer — which can be feature near tropical heat and humidity — and Winter — where weeks can go by with the daytime highs below freezing and snow piling up a meter or more. Thus, a model that predicted daily temperature would follow a sine wave with a period of 365 days and an amplitude of about 30 degrees C might show considerable skill at long term prediction while being more or less useless at predicting whether it will rain tomorrow.
Let me hasten to say that I see no reason to believe that current climate models are more reliable at predicting future climate than a magic eight ball or monkeys throwing darts.

Mindert Eiting

Suppose, we have an exam test consisting of 30 two-choice items. If you do not know anything about the subject, your expected number of correct answers is 15. Sometimes it may happen that someone answers all items correctly but also that someone does everything wrong. This may happen in a large population of subjects without knowledge. Usually, we interpret 30 items correct as a sign of knowledge. Therefore, all items false may also mean that someone did everything deliberately wrong.

Scottie

“…consisting simply of using the last period’s value in each location as the forecast for the next period’s value in that location.”

This reminds me of the standing joke about the accuracy (or otherwise) of weather forecasts amongst professional aviators. We knew that the most accurate forecast was likely to be:
“Forecast for tomorrow; the same as today. Outlook; no change.”

mt

a “random walk” alternative, consisting simply of using the last period’s value in each location as the forecast for the next period’s value in that location.

That’s not a random walk, that’s a persistence model, a.k.a a straight line.

The test measures the sum of errors relative to the random walk.

Assuming we’re talking about the persistence model above, all the test is doing is scoring the amount of change from the baseline. This says nothing about what is right or wrong. This is at best misleading to try and judge the performance of climate models using a test like this.

So much math this early?
I need more coffee.
Thank you for the information. I love Scientific Inquiry. Unfortunately, most of today’s ‘Science’ seems to be more speculative than realistic. Therefore, I like your skepticism.
At least for now.
Ghost.

Bob Ryan

No MT that is not what is assumed. With a simple random walk the last observation represents the instantaneous mean of a probability distribution (at the simplest a normal distribution) of known variance from which the next observation will be randomly drawn.

mt

BTW, the paper is referenced is:
Fildes, R. and N. Kourentzes, 2011: Validation and forecasting accuracy in models of climate change. International Journal of Forecasting. doi 10.1016/j.ijforecast.2011.03.008
and is available as a PDF here

Ian W

Letmethink says:
June 14, 2012 at 2:25 am
@ Roger Longstaff
“Is there anybody here who understands how numerical models can be “low skill” over short timescales but could possibly be accurate enough to justify massive intervention over multi-decadal timescales?”
No, because this is not remotely feasible. The climate is a chaotic system so a tiny variation in an initial condition will result in entirely unpredictable (or random) outcomes.

The argument is:::
A model could show it would not rain in UK in June on the 12th to 15th but on all other days it would – whereas in reality it did not rain on the 22nd to the 25th but on all other days it did. There was no ‘skill’ in the forecast of the 12th-15th or 22nd-25th – but for the overall month of June the model said it rained 26 days out of 30 and that could be claimed to be high skill in aggregate.
However, the longer a model runs without showing any skill the less likely it is to be skillful in aggregate and at some point it can be said that there is no chance of recovery. Effectively, this is the weather vs climate argument in different clothes.

Hey, no fair, that plot looks almost like my baseball “runnings,” graphical displays of MLB’s standings over the year. For a while the AL East was threatening to have all the teams come back to 0 games from 0.500 they had at the start of the season!
See http://wermenh.com/runnings_2012.html
It would be fun to take the lines in the plot above and match them to different baseball teams.

Ian W

I wonder if anyone can take this further and show that the random walk data put through Michael Mann’s analysis will always show a ‘hockey stick’?

Steve M. from TN

@mt,
I think you misunderstand what “random walk” is, and you totally ignore the graph at the top of the post.

Mindert Eiting says: June 14, 2012 at 4:23 am
Suppose, we have an exam test consisting of 30 two-choice items. If you do not know anything about the subject, your expected number of correct answers is 15.
— — —
I just so happened to read just hours ago an article by William Briggs on this subject of what we mean by “guessing by chance”.
http://wmbriggs.com/blog/?p=3583

TRBixler

Low skill has not caused the Obama administration to back off of its AGW agenda. Skyrocket energy costs because the computer told me to do it.

Steve M. from TN

@mt,
Oh, if you’d like to do your own random walk graph, and have Excel, try this:
in cell A1, put =randbetween (-1,1)
in cell A2, put =randbetween(-1,1)+A1
use the fill option and fill the A2 cell down a few hundred cells.
select the cells and make a line graph.
press f9 to refresh, and you’ll get quite a few graphs that look a lot like temperature graphs.

Alan D McIntire

William Briggs had an interesting article on this here:
http://wmbriggs.com/blog/?p=257
I just came across it recently. The article motivated me to read up on the “arcsine” rule,
to go into r-project.org and download the R program, learn something about computing regression using R, and to have some fun running and rerunning Wiliam Briggs’ fun program. ,

A fan of *MORE* discourse

Tamsin, please let me offer brief comment that relates both to your weblog’s post “All Models are Wrong: Limitless Possibilities” and to Anthony Watt’s present headline post on WUWT titled “WUWT: Climate Models Outperformed by Random Walks”.
As a warm-up, let’s consider a non-controversial subject: models of turbulent flow over airframes. As we improve the spatial and temporal resolution of airflow simulations, we find that our ability to predict microscopic details of the flow does *not* improve. The reason is simple: the microscopic dynamics are chaotic, such that no dynamical model (however sophisticated) can predict their future evolution with microscopic accuracy.
None-the-less, experience shows that fluid dynamical simulations DO successfully predict (typically within errors of order one percent) the flight characteristics that we mainly care about, including (for example) fuel efficiency, stall-speeds, and g-limits.
How does this happen? It happens because the microscopic dynamics is governed by global conservation laws and thermodynamic constraints, chief among them being strict conservation of mass and energy and global increase in entropy. So instant-by-instant, we don’t know whether a Karman vortex will spin-off an extended wing-flap, and yet minute-by-minute, we can predict the lift, drag, and glide-path of an airliner with considerable accuracy and confidence.
As with fluid dynamics, so with climate dynamics. Chaotic fluctuations on continental spatial scales and decadal time scales are difficult to predict with confidence. Yet global climate changes are constrained by strict conservation of mass and energy and global increase in entropy, and thus *CAN* be predicted. So year-by-year, we don’t know whether the local weather will be hot or cold, and yet decade-by-decade, we can predict the warming of the earth, and the rise of the sea, with considerable accuracy and confidence.
Appreciating this, James Hansen and his colleagues have focussed their predictions on the global energy balance, and in particular, upon sea-level rise as an integrative proxy for that global energy balance. In 2011 they confidently predicted an acceleration in sea-level rise for the coming decade. Hansen’s prediction required a certain measure of scientific boldness, since at the time satellites were showing a pronounced decrease in the sea-level rise-rate.
In coming years we will see whether Hansen’s prediction is correct. Supposing that the prediction *is* proved correct, then the concerns of rational climate-change skeptics will be largely addressed.
More broadly, it is global conservation of mass and energy and global increase in entropy that explain why simulations of both airflow and climate can be inaccurate on individual space-time gridpoints, yet accurate globally.
—————————
LOL … for fun, I’ll post this essay to Judith Curry’s Climate Etc. and to Tamsin Edwards’ All Models are Wrong too.
It will be fun to see which forums are mainly interested in rational arguments accompanied by scientific links, versus mainly interested in compiling an ideology-first “enemy list.”   🙂

Philip Richens

Roger Longstaff @ June 14, 2012 at 4:12 am
Hi Roger,
I don’t think the models perform any specific filtering between calculation steps. They *simply* solve the discretized differential equations to obtain values over a 3D grid. Important processes like clouds occur within a grid unit, and are represented in the model using heuristics. Such processes may therefore not be realistically modelled, especially over longer time-scales. Is this the kind of issue you are thinking about? Another possible reason for low decadal variability is that some important internal processes are not represented at all.
Whatever the reason, models do produce decadal variability that is low compared with measured values. If this were not the case, then the short term random fluctuations would cancel out on the average and Richard’s argument would be more reasonable.
You also wondered whether you were putting the right question to the MO. I think it was a great question you asked, but for myself I’d be rather grateful simply to hear whatever explanation the MO cares to offer about the low decadal variability. I think this is a key question.

RockyRoad

mt says:
June 14, 2012 at 4:47 am


Assuming we’re talking about the persistence model above, all the test is doing is scoring the amount of change from the baseline. This says nothing about what is right or wrong. This is at best misleading to try and judge the performance of climate models using a test like this.

Ah, the tussle between “right” and “wrong”.
If you read closer, the “RW” (for Random Walk) models had less error (E) in predicting what actually happened (“measured”) than ideology-driven “Climate Models”. So ERM < ECM.
This does indeed say which is (more) right and which is (more) wrong. That is, if your primary concern when building models in the first place is to determine which actually MODELS.
But for those still unwilling to grasp the concepts or have hockey sticks for spines, continue singing (to any tune you like):
“Don’t bother me with reality; what I’m looking for is a good fantasy.”

Jer0me

Shevva says:
June 14, 2012 at 3:13 am

‘the financial status of a gambler ‘, I spend 4 pounds a week on the lottery and always lose

yeahbut … National Lotteries are a tax on people who cannot do mathematics.
NO sarc….

ferdberple

Philip Richens says:
June 14, 2012 at 2:27 am
Pretty certain that what RB has in mind is that (A) short term random noise (white noise) prevents models from making accurate forecasts over weeks and months and (B) the white noise is averaged out over climate time scales (decades)
=========
Correct, the climate models make the assumption that climate is normally distributed (constant mean and deviation), that the errors plus and minus average out over time.
However, what if climate models function more like an inertial guidance system? The models are telling us that temperature will go up and down each day, but there is a small error in this calculation, so that slowly but surely the models will drift away from reality.
This is what happens in a guidance system. Without some external reference point, like a satellite or radio fix, airplanes, boats and spacecraft drift off course. The errors do not even out, no matter how powerful the computers, no matter how expensive the technology. There is no know solution to this problem.
The problem is that there is no external reference point for predicting the future. No satellite or radio to tell us where the future lies. You can use history to improve hind casting, but no matter how good your computers, they will slowly drift off course when steering us into the future.
This is the problem completely overlooked by climate science and the IPCC in their attempt to create an “ensemble” of forecasts to correct for this problem. No matter how many computers you use in a guidance system, it will still drift away from reality without an external reference point, and there is no reference point for the future.

Hi Anthony. Since the purpose of these models is not to predict climate or temperatures so much as change human behavior it makes since a random walk would be more predictable.
But the schemers are just getting cranked up. I just finished my story on the Belmont Forum and the Belmont Challenge being managed by the US NSF and UK’s NERC. Lots of plans on how to make decadal models that will reliably predict human behavior.
http://www.invisibleserfscollar.com/the-belmont-challenge-and-the-death-of-the-individual-via-education/
I have the actual White Paper as well as the downloaded Earth System Analysis and Prediction System. I did not provide links because they will be off the servers immediately. But that is what I am quoting from.
There is no individual human freedom left in the kind of “coupled human-environmental modeling framework” this informal lobby is pushing.

A fan of *MORE* discourse says:
June 14, 2012 at 6:22 am
As with fluid dynamics, so with climate dynamics.

False analogy.
Chaotic fluctuations on continental spatial scales and decadal time scales are difficult to predict with confidence. Yet global climate changes are constrained by strict conservation of mass and energy and global increase in entropy, and thus *CAN* be predicted.
You just need a program sophisticated enough to account for all the variables — which hasn’t been written yet — and a computer powerful enough to run it — which hasn’t been built yet.
So year-by-year, we don’t know whether the local weather will be hot or cold, and yet decade-by-decade, we can predict the warming of the earth, and the rise of the sea, with considerable accuracy and confidence.
Sooooo, how do you explain the divergence between predicted and observed this past decade?

jayhd

Will Nitschke @3:08 am,
Will, the models don’t work in actuality either.
Jay Davis