Guest post by Dr. David Stockwell
In Australia, the carbon-tax juggernaut rolls on, justified in part by fear of droughts, increasing in frequency and severity as CO2 increases.
I have always found that checking one’s assumptions was good advice, and with that in mind I checked the models used in a major drought study by the CSIRO and the Australian BoM. The study, the Drought Exceptional Circumstances Report (DECR), was widely used to support the contention that major increases in drought frequency and severity in Australia will result from further increases in CO2.
The results were recently published in the peer-reviewed journal Energy and Environment (PDF).
My paper contributes in the areas of validation of climate models (the subject of a recent post at Climate, Etc.), and regional model disagreement with rainfall observations (see post by Willis Eschenbach).
Specifically, droughts have decreased last century in line with increasing rainfall, but the climate models used in the DECR showed the opposite (and significantly so).
Overall, it is a case study demonstrating the need for more rigorous and explicit validation of climate models if they are to advise government policy.
It is reasonably well known that general circulation models are virtually worthless at projecting changes in regional rainfall, the IPCC says so, and the Australian Academy of Science agrees. The most basic statistical tests in the paper demonstrate this: the simulated drought trends are statistically inconsistent with the trend of the observations, a simple mean value shows more skill that any of the models, and drought frequency has dropped below the 95% CL of the simulations (see Figure).
The larger issue is how to get people to accept that there will always be worthless models, and the task of genuinely committed modellers to identify and eliminate worthless models. It’s not convincing to argue that validation is too hard for climate models, they are the only ones we have got, they are justified by physical realism, or they are ‘close enough’.
My study shows that the obvious testing regimes would have shown the drought models in the DECR study were unfit for use, if they had been tested. I asked CSIRO, but no validation results were supplied.
The concerns of scientists are different to decision-makers. While scientists are mainly interested with the relative skill of models in order to gauge improvements, decision-makers are (or should be) concerned primarily with whether the models should be used at all (are fit-for-use). Because of this, model-testing regimes for decision-makers must have the potential to completely reject some or all of the models if they do not rise above a predetermined standard, or benchmark.
There are a number of ways that benchmarking can be set up, which engineers or others in critical disciplines would be familiar with, usually involving a degree of independent inspection, documentation of expected standards, and so on. My favorite benchmark test is quick and easy: the Nash-Sutcliffe Efficiency, an indicator of whether a model shows more skill than a simple mean value.
I believe that decision-makers should not take results seriously unless rigorous validation of the models is also demonstrated.
It is up to the customers of climate studies to not rely on the authority of the IPCC, the CSIRO and the BoM, and to demand “Show us your tests”, as would be expected with any economic, medical or engineering study where the costs of making the wrong decision are high. Duty of care requires confidence that all reasonable means have been taken to validate all of the models and assumptions that support the key conclusions.
===============================================================
About the author (from his website here)
After receiving a Ph.D. in Ecosystem Dynamics from the Australian National University in 1992, I worked as a consultant (WHO, Parks and Wildlife, Land and Natural Resources services) until moving to the San Diego Supercomputer Center at University of California San Diego in 1997. There I helped to develop computational and data intensive infrastructure for ecological niche modeling mainly using museum collections data with grants from the NSF, USGS and DOT. I developed the GARP (Genetic Algorithm for Rule-set Production) system making contributions in many fields: modeling of invasive species, epidemiology of human diseases, the discovery of seven new species of chameleon in Madagascar, and effects on species of climate change. I have published in major journals and was judged by the US Immigration Service as an Outstanding Researcher, recognized internationally as outstanding in their academic field.
Looking at the graphs reminds me of a deeply philosophical folk song from the Australian outback:
Kookaburra sits on electric wire,
jumping up and down with their feet on fire,
burn kookaburra, burn kookarra,
how tasty you will be!
The kookaburra represents the soul of the native Aborigine who’s way of life is being threatened by the intrusion of the telegraph wires. This fourfold hermeneutic sets the cultural norm of the “us” against the animalistic spirituality of the them and a ritual of almost canabalistic consumption.
Which is all utter carp … I’m trust parodying the nonsense that some people seem to draw out of wiggly lines.
And by the way, you can’t burn on an electric wire … not unless you are straddling them!
This is an important paper.
Validation is always avoided in climate ‘science’ where a scatter shot approach is taken so there is always one model that can be quoted to match one issue (although not all of them). It was validation failure that led to the ‘hide the decline’ trick: which was more correctly fudge the results of a validation failure and claim validity. In many areas this would be seen as unscientific fraudulent behavior a dismissable offense and in some even criminal: in climate ‘science’ tricks such as this to avoid showing validation failures appear to be normal practice. (Or so it was said in defense of the climategate emails)
So by all means ask for validation of climate models – but unlike any other area of science there will never be a formal independent validation of a climate model against the real world. There hasn’t been in the past.
The best approach may be to point out the validation failures to politicians and tell them that they are making decisions based on models that are demonstrably incorrect. AND therefore, that those politicians will be held accountable for making decisions based on information that they KNOW to be incorrect as they have now been informed
They _might_ take notice.
I read a comment from some AGW academic in Australia who stated something like, “the science of humans causing climate change is unequivocal, and its going to get drier over the next century”. A few weeks later Queensland flooded.
I do wish they would separate what they consider ‘unequivocal’, and what isn’t, whether Qld gets drier or wetter certainly isnt certain, even if their AGW theories are correct.
Models, models everywhere and nary a fact in sight…..
Oz will need more than a wizard to release them from the [CO2] spell.
Julia’s carbon tax isn’t based on science. It’s a combination of need for revenue and need to keep the greens in the government. As I understand it, this tax is being driven by greens and the Labor government has to go along or be forced into a new election.
Mike, FYI in Africa do you know at least when was the major drought happening? Last interglacial Holocene optimum 5,000BP (warm period) or during the preceeding glaciation ?
Model ?
And pray Sir, where is your certificate ?
( only certificates issued by a ‘Cray XT5’ ( 2500TFLOPS or better ) will be considered )
writeyourownprogram.com @intel inside.
Good model – yet not simple enough for the public at large to grasp. Which is why the AGW side just rolls on – they know who they are dealing with on a regular basis. The Australian example for their policy is simple, they rely on an uninformed public necessary for governments to operate in the field of human behavior. If there is one thing politicians, journalists, and TV talking heads, few if any with a science background, know all too well put it out there and repeat until the public turns blue in the face. They all understand they have three sets of dedicated believers – those who rely on governments for all their information, those who believe in AGW regardless, and the vast majority of the average graduating high school public with no science at all and will never engage in lifelong learning.
Skeptics like us use “science” to argue our point of view which for the most part are like talking to dogs – they only understand their name and a few other words the rest is pure babble. Therefore the question is, how do we win this fight in language the public instinctive understands. Sagan for instance made science cool and he brought you along for the ride – he taught you as he went along. Today’s scientists with the exception of Jones, Hansen, IPCC etc., who did a bang up job with the public, are seemingly incapable of communicating with the public at all. The hockey stick whether you like it or not was brilliant strategy.
No doubt we can easily win on science – but how do we win with politicians? With journalists? With TV talking heads? In turn the public? And this is where this fight must be won – with the public. Suggestions I can use?
steveta_uk: Because extremes are defined relative to a fixed value, the A graph representing extremes of temperature increases exponentially with temperature increase, even though the temperature increases roughly linearly in Australia. You reach the point where temperature is so far above last centuries average that the indicator is extreme all the time. It shows (1) the models do match temperature (anomalies) over last century, and (2) that the benchmarks I have used work when the models do in fact match observations.
But this is about drought, the models more or less match temperature, but do a really lousy job on rainfall. Because rainfall in Australia increased last century and the models predicted a decrease, the drought index is also the opposite of reality.
Mike: I know about E&E and that it will be an excuse for the paper to be ignored. My experience with them was good, the referees were thorough, though I expect the way they operate is to choose ones that erect impossible hurdles because its anti-AGW. The choice is to get it published there, or waste my time and get frustrated elsewhere.
Wil: My audience is the policy-maker, decision-maker whatever you call them. What could be simpler than demanding models show explicit validation? People understand models are all unreliable, in econometrics, everywhere.
I get the impression that the regional planning people are now moving away from models and just talking in terms of “if temps go up 1C then do this, if 2C do that” rather than trying to downscale the climate models.
Mike: “Look’s like you made the same mistake Eschenbach acknowledged making: drought is not just a function of rainfall but of soil moisture.” The drought model in DECR is based on the rainfall, and so if that is wrong then the soil moisture model is wrong too. The DECR does look at the soil model, but since I show that a major input to it is wrong, there is no point. I don’t have to deal with definitions of drought, as the purpose is to test what was done in the DECR.
etudiant: “It also indicates that areas of extreme drought are declining, contrary to the modeled trend. On balance, if these two graphs are representative, the models are doing a fair job, imo. So what is the issue?”
You sound like a climate scientist. (I had the exact same comment from a reviewer at the AMM.) The models of drought are (1) worse that a random guess, (2) trend completely the wrong way (significantly). So any extrapolation into the future, and hence and decision you make, will probably be completely wrong. That might be good enough for a tenured academic or civil servant, but landowners and investors go broke making decision like that.
Nick Bentley: I have read of drought being described as a reduction in lateral transport of water, and that may be what underlies the increase in predicted drought. If winds decrease in models, then rain could decrease over land, even though rain increases over water.
Mervyn: “I have not yet heard anyone praise the computer models and their predictions, projections, forecasts or whatever one wants to to call their “garbage out”!”
I don’t really have a problem with using models if they are used correctly.
Katherine: “Is the handful of years when the drought frequency dropped below the 95% CL of the simulations that significant?” Yes, but as a test that does not seem very rigorous, so I don’t highlight it.
Last time I checked a warmer world was wetter but maybe I live in a different reality?
Initially it all started out as a “carbon tax” with the PM saying “I don’t mind calling it a carbon tax”. However, has anybody else noticed that with all the backlash and plummeting polls for dear Julia how this wording has suddenly morphed into the more benign “carbon price” in juxtaposition with the words “tax cut”. This wording also suddenly appeared at the same time at in the media with ABC News in particular the same night also starting to call it a “carbon price”.
Don’t be fooled this is just a money merry-go-round-wealth distribution in the true tradition of the Labour party. It has sweet FA to do with “global warming action”.
David, an excellent post. Verify the models against reality? Shocking idea, I say, shocking. Well done.
Regarding E&E, I have had both peer-reviewed and opinion pieces published there. I found their peer review thorough. The idea that their peer-review had been “called into question” by RealClimate made me snort coffee up my nose … yeah, RealClimate is such an authority on scientific behavior …
Given the recent publication of the two laughable modeling papers in “Nature”, along with the revelations in the CRU emails of corruption of the peer review process, and the Hockeystick farrago, and the “Jesus Paper”, and the endless recycling of re-warmed “Hock-alike” studies, and the “Nature trick”, and all the rest, I’m surprised that anyone thinks that somehow E&E peer review is worse than Nature or Science peer-review … the appropriate aphorism here seems to be “Physician, heal thyself”.
w.
Thanks Willis, much appreciated. Your point about “Nature” – when they get published, statistically incorrect papers, even if there are adverse published comments, they are still heavily cited (certain Rahmstorf papers in mind there), when they provide grist for the AGW mill. So a high citation rate is no guide either.
I am working on another prominent paper now, and affirms more than ever I find I cannot trust what I read in the abstract (or the body) unless I check it myself. But there are so few people interested or skilled in model validation, particularly validation in climate science. People seem to drop 20 IQ points when they start working on climate science.
Medical science has review journals, so some kind of special “statistical reviews in climate science” would be the only way of dealing with this problem.
Re David Stockwell says:
March 24, 2011 at 2:15 pm
“So a high citation rate is no guide either.”
I have speculated that one could maximise a citation index by writing the worst paper that you could get published.
This would likely get cited by those in the field and “Me-Tooers” in other fields to show that they’d heard about it
If you would allow a slight digression Anthony, you should ask;
What is going on between at the Australian Government and the CSIRO?
The CSIRO’s and Australia’s Chief scientist Professor Penny Sackett has just recently resigned half way through her 5 year term. Although she was an AGW advocate, from the little I saw of her, I thought she was reasonably fair and encouraged scientific debate.
What I find incredible is that she was never asked to provide advice to the Government prior to the Copenhagen Climate Conference and even more astounding that she has not even met Prime Minister Gillard let alone being consulted for any input prior to Gillard announcing a carbon tax.
http://www.smh.com.au/national/chief-scientist-not-invited-to-brief-pm-senate-hears-20110223-1b5o1.html
This is contrary to The Chief Scientist’s own government website charter which stated:
“The Chief Scientist for Australia provides high-level independent advice to the Prime Minister and other Ministers on matters relating to science …”
(Found in Google summary but not now on recently changed website)
http://www.chiefscientist.gov.au/
This follows the resignation of environmental economist Dr Clive Spash who resigned in 2009 after his paper criticising Emissions Trading was allegedly censored after Government interference:
http://www.smh.com.au/environment/scientist-quits-csiro-amid-censorship-claims-20091203-k8vb.html
We often criticise the CSIRO but you may well ask how is the present government going to consult widely with the scientific community on climate change if it censors papers contrary to its policy and does not deal properly with its own scientific body.
Just a note of caution, the exceptional circumstances declaration is ultimately a political call.
I have a cattle station (ranch, for our American friends) in far south-western Queensland our rainfall records date back to 1889.
The median rainfall for that period is 325 mm.
The highest thirty-year median is 1971 – 2000 @ur momisugly 430 mm.
The lowest thirty-year median is 1922 – 1951 @ur momisugly 250 mm.
Scottish Sceptic.
Sounds more like a deeply philosophical folksong, from the university campus dope smoking group.
“Mike says:
March 24, 2011 at 6:51 am
I just did a very quick read of your paper. Look’s like you made the same mistake Eschenbach acknowledged making: drought is not just a function of rainfall but of soil moisture. The did well on temperature. Higher temps generally lead to quicker evaporation. The report you are criticizing says:
“Projected increases in the areal extent and frequency of exceptionally low soil moisture years are slightly clearer than those for rainfall. If soil moisture were the sole criterion for EC declarations, then the mean projections indicate that more
declarations would be likely by 2030, particularly in the SW, SW WA and Vic&Tas regions. Under the high scenario, EC declarations would be triggered almost twice as often in most regions and almost four times as often in SWWA.”
http://www.bom.gov.au/climate/droughtec/
But again I have not read your work carefully as I need to get ready for work now. I hope your efforts get the consideration they are due.”
Mike, you are wrong and your snide comment at the end is unwarranted; in respect of temperature and evaporation you should read Franks:
http://www.agu.org/pubs/crossref/2009/2009GL040598.shtml
http://www.australianclimatemadness.com/2009/11/shock-murray-darling-warming-not-due-to-co2/
Thing is — there is little correlation between the various maxima and minima. You can have a low maximum yet have a high minimum or a low minimum but a high maximum. It is really only the minimum that means anything, as far as I am concerned.
Joe Lalonde says:
March 24, 2011 at 6:00 am
We don’t have to imagine. The AGW crowd give us samples every week.
most Australians don,t trust the CSIRO they are a arm of the labour party I feel sorry for all the good scientists who have to keep there mouth shut, OR they will face the SACK and the GOVT will cut there funding .one day somebody will have the guts to speak out . IT BETTER HAPPEN BEFORE THE GREENS TAKE HAV BALANCE OF POWER NEXT YEAR AFTER THAT WE WILL BE ALL WEAVING BASKETS UNDER A TREE