Guest post by Dr. David Stockwell
In Australia, the carbon-tax juggernaut rolls on, justified in part by fear of droughts, increasing in frequency and severity as CO2 increases.
I have always found that checking one’s assumptions was good advice, and with that in mind I checked the models used in a major drought study by the CSIRO and the Australian BoM. The study, the Drought Exceptional Circumstances Report (DECR), was widely used to support the contention that major increases in drought frequency and severity in Australia will result from further increases in CO2.
The results were recently published in the peer-reviewed journal Energy and Environment (PDF).
My paper contributes in the areas of validation of climate models (the subject of a recent post at Climate, Etc.), and regional model disagreement with rainfall observations (see post by Willis Eschenbach).
Specifically, droughts have decreased last century in line with increasing rainfall, but the climate models used in the DECR showed the opposite (and significantly so).
Overall, it is a case study demonstrating the need for more rigorous and explicit validation of climate models if they are to advise government policy.
It is reasonably well known that general circulation models are virtually worthless at projecting changes in regional rainfall, the IPCC says so, and the Australian Academy of Science agrees. The most basic statistical tests in the paper demonstrate this: the simulated drought trends are statistically inconsistent with the trend of the observations, a simple mean value shows more skill that any of the models, and drought frequency has dropped below the 95% CL of the simulations (see Figure).
The larger issue is how to get people to accept that there will always be worthless models, and the task of genuinely committed modellers to identify and eliminate worthless models. It’s not convincing to argue that validation is too hard for climate models, they are the only ones we have got, they are justified by physical realism, or they are ‘close enough’.
My study shows that the obvious testing regimes would have shown the drought models in the DECR study were unfit for use, if they had been tested. I asked CSIRO, but no validation results were supplied.
The concerns of scientists are different to decision-makers. While scientists are mainly interested with the relative skill of models in order to gauge improvements, decision-makers are (or should be) concerned primarily with whether the models should be used at all (are fit-for-use). Because of this, model-testing regimes for decision-makers must have the potential to completely reject some or all of the models if they do not rise above a predetermined standard, or benchmark.
There are a number of ways that benchmarking can be set up, which engineers or others in critical disciplines would be familiar with, usually involving a degree of independent inspection, documentation of expected standards, and so on. My favorite benchmark test is quick and easy: the Nash-Sutcliffe Efficiency, an indicator of whether a model shows more skill than a simple mean value.
I believe that decision-makers should not take results seriously unless rigorous validation of the models is also demonstrated.
It is up to the customers of climate studies to not rely on the authority of the IPCC, the CSIRO and the BoM, and to demand “Show us your tests”, as would be expected with any economic, medical or engineering study where the costs of making the wrong decision are high. Duty of care requires confidence that all reasonable means have been taken to validate all of the models and assumptions that support the key conclusions.
===============================================================
About the author (from his website here)
After receiving a Ph.D. in Ecosystem Dynamics from the Australian National University in 1992, I worked as a consultant (WHO, Parks and Wildlife, Land and Natural Resources services) until moving to the San Diego Supercomputer Center at University of California San Diego in 1997. There I helped to develop computational and data intensive infrastructure for ecological niche modeling mainly using museum collections data with grants from the NSF, USGS and DOT. I developed the GARP (Genetic Algorithm for Rule-set Production) system making contributions in many fields: modeling of invasive species, epidemiology of human diseases, the discovery of seven new species of chameleon in Madagascar, and effects on species of climate change. I have published in major journals and was judged by the US Immigration Service as an Outstanding Researcher, recognized internationally as outstanding in their academic field.
I seems funny that some years ago CSIRO stated that models were to be viewed with caution because they relied too much on assumptions that were not always correct. The same applies today especially with climate models.
thanks!
we KNOW the Bom and CSIRO LIE!
we need info like this to enable us to fight back.
problem is, ignorant pollies who wont read anything but what their agenda( for revenue raising ie JuLiars Carbon Tax ) allows ie Ipcc drivel.
I have not yet heard anyone praise the computer models and their predictions, projections, forecasts or whatever one wants to to call their “garbage out”! When companies put out information to the public, it must be audited or be subjected to some form of official assurance review to minimize the risk of directors engaging in misleading and deceptive conduct. I cannot understand why similar standards cannot apply to model-based predictions… particularly since the IPCC so heavily relies on them and governments rely on the IPCC in formulating their climate change policies.
Based on the report ‘Carbon Dioxide and Earth’s Future: Pursuing the Prudent Path’ by Craig Idso and Sherwood Idso (see link below), it would seem that model-based predictions are seriously flawed and unreliable, which means the IPCC is wrong, which means governments forming policies based on the IPCC’s report must be flawed.
http://www.co2science.org/education/reports/prudentpath/prudentpath.php
Well well well…
“It is up to the customers of climate studies to not rely on the authority of the IPCC, the CSIRO and the BoM, and to demand “Show us your tests”, as would be expected with any economic, medical or engineering study where the costs of making the wrong decision are high. ”
In the case of climate studies, they should say “show independent scientists you raw data, our data manipulation methods and your tests”. I wouldn’t trust what hockey-stick manufacturers showed me.
Does their rainfall model include the Indian Ocean Dipole?
http://www.science.unsw.edu.au/news/indian-ocean-drought/
By the way, has anyone yet determined drivers for the IOD?
Fine essay as always Mr Stockwell! We have a carbon Tax here in my neck of the woods, and it might be indicative of what you will soon be experiencing. The tax has merely ended up paying for re-election campaigns or funnelled to radical advocacy groups. It has had zero effect on consumption and even less effect on emissions.
I’ll try and dig up some data for you.
Cheers!
I think that the idea of validating models, while novel, is an idea that should be considered in the future.
Can we have a bit more explanation of “A” please?
To me, it looks like exponential growth in area experiencing exceptional temperatures prior to 2000, so presumably reached 100% a few years back. Is this what it is supposed to mean?
As always it is not about the science it is about the agenda. Government is in control and wants to assert more control (read taxes), or dreams on an ideal world (read sycophant) .
In the UK attenuated storm water storage systems and storm water drainage systems are required to be designed for a 1 in 100 year storm plus 30% for climate change?
This results in the additional use of materials, transport and labour etc, which of course is very sustainable indeed? The inmates are defintaly running the lunatic asylum.
A bit of rain in Victoria of late. It will be interesting to see how they explain that. Personally, it is standard for a La Nina event.
Dr. Stockwell,
I commend your efforts at looking at bad science practices.
It is far easier to ignore current readings and cherry pick or manipulate whatever for the sake of more funding at the expense of good science.
Can you imagine the crap science would have us believe if people did not check their reports or studies?
Being just an interested observer and too lazy to go look up answers to my own questions, please forgive me for asking: Is A a striking example of another kind of hockey stick? The red line is observation of areas with exceptionally high temperatures rising drastically and off the graph starting in 1970 through 2000? I don’t understand and the post doesn’t seem to offer an explanation.
E&E is not a well regarded journal. Their peer review process has been called into question. I cannot comment on your work. Maybe it is valid. But you might have had more impact outside the blog-sphere had you submitted to a higher quality journal.
Sorry O/T, but I can’t post in the Tips & Notes for some reason. Please delete this if you wish.
The IPad problem is due to the onswipe theme that wordpress have introduced. It has made WUWT un-usable on my IPad. I believe that you can disable it, and I would ask that you consider doing this. I also notice it has infected Climate Audit and Climate Etc.
Brian Andrews
Basic science:
For a scientific theory to be valid it must:
– Explain available data
– Make predictions
– Be falsifiable
While some aspects of “Climate Theory” conform, there are too many which do not. Models are predictions, if the models do not match the observations then the models are wrong and the theory itself needs to be modified.
I just did a very quick read of your paper. Look’s like you made the same mistake Eschenbach acknowledged making: drought is not just a function of rainfall but of soil moisture. The did well on temperature. Higher temps generally lead to quicker evaporation. The report you are criticizing says:
“Projected increases in the areal extent and frequency of exceptionally low soil moisture years are slightly clearer than those for rainfall. If soil moisture were the sole criterion for EC declarations, then the mean projections indicate that more
declarations would be likely by 2030, particularly in the SW, SW WA and Vic&Tas regions. Under the high scenario, EC declarations would be triggered almost twice as often in most regions and almost four times as often in SWWA.”
http://www.bom.gov.au/climate/droughtec/
But again I have not read your work carefully as I need to get ready for work now. I hope your efforts get the consideration they are due.
I emphatically agree. There are so very many ways to get a model wrong. The only way to tell what it is worth for decision-makers is to show that it works. It has to work when things are going up, and it also has to work when things are going down. A badly broken model can still look OK as long as the wind is blowing in one direction, the preferred direction. But when the wind changes, we find out how well it really works.
As long as la Niña continues they should worry about floodings instead….then politicians are the only ones to experience drought…of votes.
I tried to post this in Tips and Notes, but it appears to be bunged up. Stats Canada, a government agency here in Canada known for political bias has announced that the climate models are correct.
http://cnews.canoe.ca/CNEWS/Environment/2011/03/23/17728111.html
Pretty bold language! I smell a big fat civil-serpent rat!
Cheers!
Dr David Stockwell shows good sense in sugesting that consumers of the results of climate models should know that those models actually work; what alarms me is that most politicians who are insisting that ‘the climate models show us…’ would not have any idea if the models have any p[redictive value or not, but want to plunge the developed world into a huge retrograde step on faith alone.
I’m enough of a peasant to insist that the slaesman shows me his product works.
The Hockey Stick lives! 😛
Okay, in a more serious vein, a bit more discussion of the provided graphs would be nice. Is the handful of years when the drought frequency dropped below the 95% CL of the simulations that significant? Though I suppose assuming an constant 5% in area with exceptionally low rainfall would work just as well.
Interesting. I’d read that many climate models predict an increase of total precipitation, but that precipitation will also become less evenly distributed over time. That is to say that average precipitation increases, but you also get more periods of drought interspersed with large downpours. People concerned about crop yields in a warming world often point to this prediction as worrisome, since crops are easily decimated under such conditions.
My questions:
Does the model in question have anything to say about the distribution of precipitation over time?
This model predicts a decrease in the average precipitation over time, which appears to be at odds with other climate models. Why is it different?
What does the graph mean and how does the graph relate to the text?
The presentation is not directly referencing the graphs.
At first glance, the graph suggests an exponentially increasing percentage of Australia is experiencing extreme heat and that this is as indicated by the various models.
It also indicates that areas of extreme drought are declining, contrary to the modeled trend.
On balance, if these two graphs are representative, the models are doing a fair job, imo. So what is the issue?
Yes, models may be lousy, but surely sacrificing the economy will lower global temperature. Any politician knows that much.
Looking at the graphs reminds me of a deeply philosophical folk song from the Australian outback:
Kookaburra sits on electric wire,
jumping up and down with their feet on fire,
burn kookaburra, burn kookarra,
how tasty you will be!
The kookaburra represents the soul of the native Aborigine who’s way of life is being threatened by the intrusion of the telegraph wires. This fourfold hermeneutic sets the cultural norm of the “us” against the animalistic spirituality of the them and a ritual of almost canabalistic consumption.
Which is all utter carp … I’m trust parodying the nonsense that some people seem to draw out of wiggly lines.
And by the way, you can’t burn on an electric wire … not unless you are straddling them!
This is an important paper.
Validation is always avoided in climate ‘science’ where a scatter shot approach is taken so there is always one model that can be quoted to match one issue (although not all of them). It was validation failure that led to the ‘hide the decline’ trick: which was more correctly fudge the results of a validation failure and claim validity. In many areas this would be seen as unscientific fraudulent behavior a dismissable offense and in some even criminal: in climate ‘science’ tricks such as this to avoid showing validation failures appear to be normal practice. (Or so it was said in defense of the climategate emails)
So by all means ask for validation of climate models – but unlike any other area of science there will never be a formal independent validation of a climate model against the real world. There hasn’t been in the past.
The best approach may be to point out the validation failures to politicians and tell them that they are making decisions based on models that are demonstrably incorrect. AND therefore, that those politicians will be held accountable for making decisions based on information that they KNOW to be incorrect as they have now been informed
They _might_ take notice.
I read a comment from some AGW academic in Australia who stated something like, “the science of humans causing climate change is unequivocal, and its going to get drier over the next century”. A few weeks later Queensland flooded.
I do wish they would separate what they consider ‘unequivocal’, and what isn’t, whether Qld gets drier or wetter certainly isnt certain, even if their AGW theories are correct.
Models, models everywhere and nary a fact in sight…..
Oz will need more than a wizard to release them from the [CO2] spell.
Julia’s carbon tax isn’t based on science. It’s a combination of need for revenue and need to keep the greens in the government. As I understand it, this tax is being driven by greens and the Labor government has to go along or be forced into a new election.
Mike, FYI in Africa do you know at least when was the major drought happening? Last interglacial Holocene optimum 5,000BP (warm period) or during the preceeding glaciation ?
Model ?
And pray Sir, where is your certificate ?
( only certificates issued by a ‘Cray XT5’ ( 2500TFLOPS or better ) will be considered )
writeyourownprogram.com @intel inside.
Good model – yet not simple enough for the public at large to grasp. Which is why the AGW side just rolls on – they know who they are dealing with on a regular basis. The Australian example for their policy is simple, they rely on an uninformed public necessary for governments to operate in the field of human behavior. If there is one thing politicians, journalists, and TV talking heads, few if any with a science background, know all too well put it out there and repeat until the public turns blue in the face. They all understand they have three sets of dedicated believers – those who rely on governments for all their information, those who believe in AGW regardless, and the vast majority of the average graduating high school public with no science at all and will never engage in lifelong learning.
Skeptics like us use “science” to argue our point of view which for the most part are like talking to dogs – they only understand their name and a few other words the rest is pure babble. Therefore the question is, how do we win this fight in language the public instinctive understands. Sagan for instance made science cool and he brought you along for the ride – he taught you as he went along. Today’s scientists with the exception of Jones, Hansen, IPCC etc., who did a bang up job with the public, are seemingly incapable of communicating with the public at all. The hockey stick whether you like it or not was brilliant strategy.
No doubt we can easily win on science – but how do we win with politicians? With journalists? With TV talking heads? In turn the public? And this is where this fight must be won – with the public. Suggestions I can use?
steveta_uk: Because extremes are defined relative to a fixed value, the A graph representing extremes of temperature increases exponentially with temperature increase, even though the temperature increases roughly linearly in Australia. You reach the point where temperature is so far above last centuries average that the indicator is extreme all the time. It shows (1) the models do match temperature (anomalies) over last century, and (2) that the benchmarks I have used work when the models do in fact match observations.
But this is about drought, the models more or less match temperature, but do a really lousy job on rainfall. Because rainfall in Australia increased last century and the models predicted a decrease, the drought index is also the opposite of reality.
Mike: I know about E&E and that it will be an excuse for the paper to be ignored. My experience with them was good, the referees were thorough, though I expect the way they operate is to choose ones that erect impossible hurdles because its anti-AGW. The choice is to get it published there, or waste my time and get frustrated elsewhere.
Wil: My audience is the policy-maker, decision-maker whatever you call them. What could be simpler than demanding models show explicit validation? People understand models are all unreliable, in econometrics, everywhere.
I get the impression that the regional planning people are now moving away from models and just talking in terms of “if temps go up 1C then do this, if 2C do that” rather than trying to downscale the climate models.
Mike: “Look’s like you made the same mistake Eschenbach acknowledged making: drought is not just a function of rainfall but of soil moisture.” The drought model in DECR is based on the rainfall, and so if that is wrong then the soil moisture model is wrong too. The DECR does look at the soil model, but since I show that a major input to it is wrong, there is no point. I don’t have to deal with definitions of drought, as the purpose is to test what was done in the DECR.
etudiant: “It also indicates that areas of extreme drought are declining, contrary to the modeled trend. On balance, if these two graphs are representative, the models are doing a fair job, imo. So what is the issue?”
You sound like a climate scientist. (I had the exact same comment from a reviewer at the AMM.) The models of drought are (1) worse that a random guess, (2) trend completely the wrong way (significantly). So any extrapolation into the future, and hence and decision you make, will probably be completely wrong. That might be good enough for a tenured academic or civil servant, but landowners and investors go broke making decision like that.
Nick Bentley: I have read of drought being described as a reduction in lateral transport of water, and that may be what underlies the increase in predicted drought. If winds decrease in models, then rain could decrease over land, even though rain increases over water.
Mervyn: “I have not yet heard anyone praise the computer models and their predictions, projections, forecasts or whatever one wants to to call their “garbage out”!”
I don’t really have a problem with using models if they are used correctly.
Katherine: “Is the handful of years when the drought frequency dropped below the 95% CL of the simulations that significant?” Yes, but as a test that does not seem very rigorous, so I don’t highlight it.
Last time I checked a warmer world was wetter but maybe I live in a different reality?
Initially it all started out as a “carbon tax” with the PM saying “I don’t mind calling it a carbon tax”. However, has anybody else noticed that with all the backlash and plummeting polls for dear Julia how this wording has suddenly morphed into the more benign “carbon price” in juxtaposition with the words “tax cut”. This wording also suddenly appeared at the same time at in the media with ABC News in particular the same night also starting to call it a “carbon price”.
Don’t be fooled this is just a money merry-go-round-wealth distribution in the true tradition of the Labour party. It has sweet FA to do with “global warming action”.
David, an excellent post. Verify the models against reality? Shocking idea, I say, shocking. Well done.
Regarding E&E, I have had both peer-reviewed and opinion pieces published there. I found their peer review thorough. The idea that their peer-review had been “called into question” by RealClimate made me snort coffee up my nose … yeah, RealClimate is such an authority on scientific behavior …
Given the recent publication of the two laughable modeling papers in “Nature”, along with the revelations in the CRU emails of corruption of the peer review process, and the Hockeystick farrago, and the “Jesus Paper”, and the endless recycling of re-warmed “Hock-alike” studies, and the “Nature trick”, and all the rest, I’m surprised that anyone thinks that somehow E&E peer review is worse than Nature or Science peer-review … the appropriate aphorism here seems to be “Physician, heal thyself”.
w.
Thanks Willis, much appreciated. Your point about “Nature” – when they get published, statistically incorrect papers, even if there are adverse published comments, they are still heavily cited (certain Rahmstorf papers in mind there), when they provide grist for the AGW mill. So a high citation rate is no guide either.
I am working on another prominent paper now, and affirms more than ever I find I cannot trust what I read in the abstract (or the body) unless I check it myself. But there are so few people interested or skilled in model validation, particularly validation in climate science. People seem to drop 20 IQ points when they start working on climate science.
Medical science has review journals, so some kind of special “statistical reviews in climate science” would be the only way of dealing with this problem.
Re David Stockwell says:
March 24, 2011 at 2:15 pm
“So a high citation rate is no guide either.”
I have speculated that one could maximise a citation index by writing the worst paper that you could get published.
This would likely get cited by those in the field and “Me-Tooers” in other fields to show that they’d heard about it
If you would allow a slight digression Anthony, you should ask;
What is going on between at the Australian Government and the CSIRO?
The CSIRO’s and Australia’s Chief scientist Professor Penny Sackett has just recently resigned half way through her 5 year term. Although she was an AGW advocate, from the little I saw of her, I thought she was reasonably fair and encouraged scientific debate.
What I find incredible is that she was never asked to provide advice to the Government prior to the Copenhagen Climate Conference and even more astounding that she has not even met Prime Minister Gillard let alone being consulted for any input prior to Gillard announcing a carbon tax.
http://www.smh.com.au/national/chief-scientist-not-invited-to-brief-pm-senate-hears-20110223-1b5o1.html
This is contrary to The Chief Scientist’s own government website charter which stated:
“The Chief Scientist for Australia provides high-level independent advice to the Prime Minister and other Ministers on matters relating to science …”
(Found in Google summary but not now on recently changed website)
http://www.chiefscientist.gov.au/
This follows the resignation of environmental economist Dr Clive Spash who resigned in 2009 after his paper criticising Emissions Trading was allegedly censored after Government interference:
http://www.smh.com.au/environment/scientist-quits-csiro-amid-censorship-claims-20091203-k8vb.html
We often criticise the CSIRO but you may well ask how is the present government going to consult widely with the scientific community on climate change if it censors papers contrary to its policy and does not deal properly with its own scientific body.
Just a note of caution, the exceptional circumstances declaration is ultimately a political call.
I have a cattle station (ranch, for our American friends) in far south-western Queensland our rainfall records date back to 1889.
The median rainfall for that period is 325 mm.
The highest thirty-year median is 1971 – 2000 @ 430 mm.
The lowest thirty-year median is 1922 – 1951 @ 250 mm.
Scottish Sceptic.
Sounds more like a deeply philosophical folksong, from the university campus dope smoking group.
“Mike says:
March 24, 2011 at 6:51 am
I just did a very quick read of your paper. Look’s like you made the same mistake Eschenbach acknowledged making: drought is not just a function of rainfall but of soil moisture. The did well on temperature. Higher temps generally lead to quicker evaporation. The report you are criticizing says:
“Projected increases in the areal extent and frequency of exceptionally low soil moisture years are slightly clearer than those for rainfall. If soil moisture were the sole criterion for EC declarations, then the mean projections indicate that more
declarations would be likely by 2030, particularly in the SW, SW WA and Vic&Tas regions. Under the high scenario, EC declarations would be triggered almost twice as often in most regions and almost four times as often in SWWA.”
http://www.bom.gov.au/climate/droughtec/
But again I have not read your work carefully as I need to get ready for work now. I hope your efforts get the consideration they are due.”
Mike, you are wrong and your snide comment at the end is unwarranted; in respect of temperature and evaporation you should read Franks:
http://www.agu.org/pubs/crossref/2009/2009GL040598.shtml
http://www.australianclimatemadness.com/2009/11/shock-murray-darling-warming-not-due-to-co2/
Thing is — there is little correlation between the various maxima and minima. You can have a low maximum yet have a high minimum or a low minimum but a high maximum. It is really only the minimum that means anything, as far as I am concerned.
Joe Lalonde says:
March 24, 2011 at 6:00 am
We don’t have to imagine. The AGW crowd give us samples every week.
most Australians don,t trust the CSIRO they are a arm of the labour party I feel sorry for all the good scientists who have to keep there mouth shut, OR they will face the SACK and the GOVT will cut there funding .one day somebody will have the guts to speak out . IT BETTER HAPPEN BEFORE THE GREENS TAKE HAV BALANCE OF POWER NEXT YEAR AFTER THAT WE WILL BE ALL WEAVING BASKETS UNDER A TREE
David kindly showed me a preview of this paper after I had read the CSIRO Hennessy at al report of 2008, Drought and Exceptional Circumstances Report, DECR for short. Some other authors of the DECR have been prominent in IPCC authoring.
In essence, David took the definitions and framework of the DECR, then showed with a few short statistical tests that the paper lacked predictive skill. (His conclusions since 2008 are being confirmed).
Therefore, there is no need for bloggers here to get too nuanced about definitions and special cases. The DECR, from memory, defined 4 classes of drought, but a failure of any one class would fail the whole paper. Prior soil moisture might be important, as noted above, but it’s not the sole determinant. (For some of the earlier period, pan evaporimeter tests were shown inaccurate when it was realised that birds were splashing away in them. They now have a mesh cover.)
Example. There were drought-like conditions in some of the last decade at the mouth of the Murray River. The headwaters of the Murray-Darling are 1,500 km distant as the crow flies. Therefore, a significant drought 1,500 km away could cause drought-like conditions at the mouth. This raises a problem of correlation with temperature. Temperature where? Temperature when? (it can take several months for water to flow from headwater to mouth).
My reaction to graphs central to explaining the CSIRO predictions was unkind. The frequency of droughts in decades before the paper was written increased sharply for decades after the publication date. To me, rightly or wrongly, that’s a sign of confirmation bias.
Thanks Geoff. I am really perplexed by such criticisms. Any criticism of the definition of drought I used is a criticism of the definition used in the CSIRO and BoM DECR. The scholarly approach is to defeat your opponent on their own turf.
The other criticism is that the models are ‘close enough’. Here the central issue is that if you intend to use them to forecast the trend, they must show significant skill at modelling the trend in the past. To say they are ‘close enough’ is a sloppy, intuitive attitude that doesn’t get you anywhere in science (as you know Geoff). Both of these complains were made in peer review at the AMM.
I guess they figure that they spend enough on the studies to warrant spending something on audits and/or other checks.