First, a bit of a primer. Wikipedia describes a random walk is a mathematical formalisation of a trajectory that consists of taking successive random steps. For example, the path traced by a molecule as it travels in a liquid or a gas, the search path of a foraging animal, the price of a fluctuating stock and the financial status of a gambler can all be modeled as random walks. The term random walk was first introduced by Karl Pearson in 1905.
From the Financial Post: A 2011 study in the Journal of Forecasting took the same data set and compared model predictions against a “random walk” alternative, consisting simply of using the last period’s value in each location as the forecast for the next period’s value in that location.
The test measures the sum of errors relative to the random walk. A perfect model gets a score of zero, meaning it made no errors. A model that does no better than a random walk gets a score of 1. A model receiving a score above 1 did worse than uninformed guesses. Simple statistical forecast models that have no climatology or physics in them typically got scores between 0.8 and 1, indicating slight improvements on the random walk, though in some cases their scores went as high as 1.8.
The climate models, by contrast, got scores ranging from 2.4 to 3.7, indicating a total failure to provide valid forecast information at the regional level, even on long time scales. The authors commented: “This implies that the current [climate] models are ill-suited to localized decadal predictions, even though they are used as inputs for policymaking.”……
More here: http://opinion.financialpost.com/2012/06/13/junk-science-week-climate-models-fail-reality-test/
h/t to WUWT reader Crispin in Waterloo
Previously, WUWT covered this issue of random walks here:
Is Global Temperature a Random Walk?
UPDATE: The paper (thanks to reader MT) Fildes, R. and N. Kourentzes, 2011: Validation and forecasting accuracy in models of climate change. International Journal of Forecasting. doi 10.1016/j.ijforecast.2011.03.008
and is available as a PDF here
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
The paper is in the International Journal of Forecasting (not the Journal of Forecasting)
Fildes, Robert and Nikolaos Kourentzes (2011) “Validation and Forecasting Accuracy in Models of Climate Change International Journal of Forecasting 27 968-995.
http://www.sciencedirect.com/science/journal/01692070/27/4
Ross McKitrick’s fully referenced article is here (the journal reference is incorrect there too and in the FP article, but the footnote is correct…)
http://www.rossmckitrick.com/uploads/4/8/0/8/4808045/fp_june12_models_i.pdf
Thanks Philip & Letmethink.
My problem is that any filtering (between calculation steps), or “re-setting” of variables to preserve stability, inevitably leads to loss of information on the system.
Filtering is used to increase signal to noise ratio when there is “a priori” knowledge of the signal (for example in a radio receiver that uses a narrowband filter tuned to a specific frequency). With climate models there is only an assumption that there will be a GHG signal. Furthermore, any low pass filtering will remove information generated by the model itself, as signals must be sampled at twice their highest frequency component (Nyquist theory) in order to preserve information.
If one were to equate accurate information to Shannon entropy, it seems inevitable that climate models will deviate from reality exponentially with respect to time, as a consequence of the logarithmic nature of information theory.
Is there a flaw in my argument?
Roger Longstaff says:
June 14, 2012 at 1:03 am
I am still struggling to find an answer to my original question, and perhaps my questions to the Met Office were the wrong ones. Is there anybody here who understands how numerical models can be “low skill” over short timescales but could possibly be accurate enough to justify massive intervention over multi-decadal timescales?
=======
Roger
In my part of the world there is a dramatic difference between Summer — which can be feature near tropical heat and humidity — and Winter — where weeks can go by with the daytime highs below freezing and snow piling up a meter or more. Thus, a model that predicted daily temperature would follow a sine wave with a period of 365 days and an amplitude of about 30 degrees C might show considerable skill at long term prediction while being more or less useless at predicting whether it will rain tomorrow.
Let me hasten to say that I see no reason to believe that current climate models are more reliable at predicting future climate than a magic eight ball or monkeys throwing darts.
Suppose, we have an exam test consisting of 30 two-choice items. If you do not know anything about the subject, your expected number of correct answers is 15. Sometimes it may happen that someone answers all items correctly but also that someone does everything wrong. This may happen in a large population of subjects without knowledge. Usually, we interpret 30 items correct as a sign of knowledge. Therefore, all items false may also mean that someone did everything deliberately wrong.
This reminds me of the standing joke about the accuracy (or otherwise) of weather forecasts amongst professional aviators. We knew that the most accurate forecast was likely to be:
“Forecast for tomorrow; the same as today. Outlook; no change.”
That’s not a random walk, that’s a persistence model, a.k.a a straight line.
Assuming we’re talking about the persistence model above, all the test is doing is scoring the amount of change from the baseline. This says nothing about what is right or wrong. This is at best misleading to try and judge the performance of climate models using a test like this.
So much math this early?
I need more coffee.
Thank you for the information. I love Scientific Inquiry. Unfortunately, most of today’s ‘Science’ seems to be more speculative than realistic. Therefore, I like your skepticism.
At least for now.
Ghost.
No MT that is not what is assumed. With a simple random walk the last observation represents the instantaneous mean of a probability distribution (at the simplest a normal distribution) of known variance from which the next observation will be randomly drawn.
BTW, the paper is referenced is:
Fildes, R. and N. Kourentzes, 2011: Validation and forecasting accuracy in models of climate change. International Journal of Forecasting. doi 10.1016/j.ijforecast.2011.03.008
and is available as a PDF here
Letmethink says:
June 14, 2012 at 2:25 am
@ur momisugly Roger Longstaff
“Is there anybody here who understands how numerical models can be “low skill” over short timescales but could possibly be accurate enough to justify massive intervention over multi-decadal timescales?”
No, because this is not remotely feasible. The climate is a chaotic system so a tiny variation in an initial condition will result in entirely unpredictable (or random) outcomes.
The argument is:::
A model could show it would not rain in UK in June on the 12th to 15th but on all other days it would – whereas in reality it did not rain on the 22nd to the 25th but on all other days it did. There was no ‘skill’ in the forecast of the 12th-15th or 22nd-25th – but for the overall month of June the model said it rained 26 days out of 30 and that could be claimed to be high skill in aggregate.
However, the longer a model runs without showing any skill the less likely it is to be skillful in aggregate and at some point it can be said that there is no chance of recovery. Effectively, this is the weather vs climate argument in different clothes.
Hey, no fair, that plot looks almost like my baseball “runnings,” graphical displays of MLB’s standings over the year. For a while the AL East was threatening to have all the teams come back to 0 games from 0.500 they had at the start of the season!
See http://wermenh.com/runnings_2012.html
It would be fun to take the lines in the plot above and match them to different baseball teams.
I wonder if anyone can take this further and show that the random walk data put through Michael Mann’s analysis will always show a ‘hockey stick’?
@mt,
I think you misunderstand what “random walk” is, and you totally ignore the graph at the top of the post.
Mindert Eiting says: June 14, 2012 at 4:23 am
Suppose, we have an exam test consisting of 30 two-choice items. If you do not know anything about the subject, your expected number of correct answers is 15.
— — —
I just so happened to read just hours ago an article by William Briggs on this subject of what we mean by “guessing by chance”.
http://wmbriggs.com/blog/?p=3583
Low skill has not caused the Obama administration to back off of its AGW agenda. Skyrocket energy costs because the computer told me to do it.
@mt,
Oh, if you’d like to do your own random walk graph, and have Excel, try this:
in cell A1, put =randbetween (-1,1)
in cell A2, put =randbetween(-1,1)+A1
use the fill option and fill the A2 cell down a few hundred cells.
select the cells and make a line graph.
press f9 to refresh, and you’ll get quite a few graphs that look a lot like temperature graphs.
William Briggs had an interesting article on this here:
http://wmbriggs.com/blog/?p=257
I just came across it recently. The article motivated me to read up on the “arcsine” rule,
to go into r-project.org and download the R program, learn something about computing regression using R, and to have some fun running and rerunning Wiliam Briggs’ fun program. ,
Tamsin, please let me offer brief comment that relates both to your weblog’s post “All Models are Wrong: Limitless Possibilities” and to Anthony Watt’s present headline post on WUWT titled “WUWT: Climate Models Outperformed by Random Walks”.
As a warm-up, let’s consider a non-controversial subject: models of turbulent flow over airframes. As we improve the spatial and temporal resolution of airflow simulations, we find that our ability to predict microscopic details of the flow does *not* improve. The reason is simple: the microscopic dynamics are chaotic, such that no dynamical model (however sophisticated) can predict their future evolution with microscopic accuracy.
None-the-less, experience shows that fluid dynamical simulations DO successfully predict (typically within errors of order one percent) the flight characteristics that we mainly care about, including (for example) fuel efficiency, stall-speeds, and g-limits.
How does this happen? It happens because the microscopic dynamics is governed by global conservation laws and thermodynamic constraints, chief among them being strict conservation of mass and energy and global increase in entropy. So instant-by-instant, we don’t know whether a Karman vortex will spin-off an extended wing-flap, and yet minute-by-minute, we can predict the lift, drag, and glide-path of an airliner with considerable accuracy and confidence.
As with fluid dynamics, so with climate dynamics. Chaotic fluctuations on continental spatial scales and decadal time scales are difficult to predict with confidence. Yet global climate changes are constrained by strict conservation of mass and energy and global increase in entropy, and thus *CAN* be predicted. So year-by-year, we don’t know whether the local weather will be hot or cold, and yet decade-by-decade, we can predict the warming of the earth, and the rise of the sea, with considerable accuracy and confidence.
Appreciating this, James Hansen and his colleagues have focussed their predictions on the global energy balance, and in particular, upon sea-level rise as an integrative proxy for that global energy balance. In 2011 they confidently predicted an acceleration in sea-level rise for the coming decade. Hansen’s prediction required a certain measure of scientific boldness, since at the time satellites were showing a pronounced decrease in the sea-level rise-rate.
In coming years we will see whether Hansen’s prediction is correct. Supposing that the prediction *is* proved correct, then the concerns of rational climate-change skeptics will be largely addressed.
More broadly, it is global conservation of mass and energy and global increase in entropy that explain why simulations of both airflow and climate can be inaccurate on individual space-time gridpoints, yet accurate globally.
—————————
LOL … for fun, I’ll post this essay to Judith Curry’s Climate Etc. and to Tamsin Edwards’ All Models are Wrong too.
It will be fun to see which forums are mainly interested in rational arguments accompanied by scientific links, versus mainly interested in compiling an ideology-first “enemy list.” 🙂
Roger Longstaff @ur momisugly June 14, 2012 at 4:12 am
Hi Roger,
I don’t think the models perform any specific filtering between calculation steps. They *simply* solve the discretized differential equations to obtain values over a 3D grid. Important processes like clouds occur within a grid unit, and are represented in the model using heuristics. Such processes may therefore not be realistically modelled, especially over longer time-scales. Is this the kind of issue you are thinking about? Another possible reason for low decadal variability is that some important internal processes are not represented at all.
Whatever the reason, models do produce decadal variability that is low compared with measured values. If this were not the case, then the short term random fluctuations would cancel out on the average and Richard’s argument would be more reasonable.
You also wondered whether you were putting the right question to the MO. I think it was a great question you asked, but for myself I’d be rather grateful simply to hear whatever explanation the MO cares to offer about the low decadal variability. I think this is a key question.
mt says:
June 14, 2012 at 4:47 am
Ah, the tussle between “right” and “wrong”.
If you read closer, the “RW” (for Random Walk) models had less error (E) in predicting what actually happened (“measured”) than ideology-driven “Climate Models”. So ERM < ECM.
This does indeed say which is (more) right and which is (more) wrong. That is, if your primary concern when building models in the first place is to determine which actually MODELS.
But for those still unwilling to grasp the concepts or have hockey sticks for spines, continue singing (to any tune you like):
“Don’t bother me with reality; what I’m looking for is a good fantasy.”
Shevva says:
June 14, 2012 at 3:13 am
yeahbut … National Lotteries are a tax on people who cannot do mathematics.
NO sarc….
Philip Richens says:
June 14, 2012 at 2:27 am
Pretty certain that what RB has in mind is that (A) short term random noise (white noise) prevents models from making accurate forecasts over weeks and months and (B) the white noise is averaged out over climate time scales (decades)
=========
Correct, the climate models make the assumption that climate is normally distributed (constant mean and deviation), that the errors plus and minus average out over time.
However, what if climate models function more like an inertial guidance system? The models are telling us that temperature will go up and down each day, but there is a small error in this calculation, so that slowly but surely the models will drift away from reality.
This is what happens in a guidance system. Without some external reference point, like a satellite or radio fix, airplanes, boats and spacecraft drift off course. The errors do not even out, no matter how powerful the computers, no matter how expensive the technology. There is no know solution to this problem.
The problem is that there is no external reference point for predicting the future. No satellite or radio to tell us where the future lies. You can use history to improve hind casting, but no matter how good your computers, they will slowly drift off course when steering us into the future.
This is the problem completely overlooked by climate science and the IPCC in their attempt to create an “ensemble” of forecasts to correct for this problem. No matter how many computers you use in a guidance system, it will still drift away from reality without an external reference point, and there is no reference point for the future.
Hi Anthony. Since the purpose of these models is not to predict climate or temperatures so much as change human behavior it makes since a random walk would be more predictable.
But the schemers are just getting cranked up. I just finished my story on the Belmont Forum and the Belmont Challenge being managed by the US NSF and UK’s NERC. Lots of plans on how to make decadal models that will reliably predict human behavior.
http://www.invisibleserfscollar.com/the-belmont-challenge-and-the-death-of-the-individual-via-education/
I have the actual White Paper as well as the downloaded Earth System Analysis and Prediction System. I did not provide links because they will be off the servers immediately. But that is what I am quoting from.
There is no individual human freedom left in the kind of “coupled human-environmental modeling framework” this informal lobby is pushing.
A fan of *MORE* discourse says:
June 14, 2012 at 6:22 am
As with fluid dynamics, so with climate dynamics.
False analogy.
Chaotic fluctuations on continental spatial scales and decadal time scales are difficult to predict with confidence. Yet global climate changes are constrained by strict conservation of mass and energy and global increase in entropy, and thus *CAN* be predicted.
You just need a program sophisticated enough to account for all the variables — which hasn’t been written yet — and a computer powerful enough to run it — which hasn’t been built yet.
So year-by-year, we don’t know whether the local weather will be hot or cold, and yet decade-by-decade, we can predict the warming of the earth, and the rise of the sea, with considerable accuracy and confidence.
Sooooo, how do you explain the divergence between predicted and observed this past decade?
Will Nitschke @3:08 am,
Will, the models don’t work in actuality either.
Jay Davis