While the reactions to Lewis and Crok rage, from intellectually lazy trolling at Dr. Judith Curry’s shop by a trio of commenters using the “nyah, nyah, it’s not peer reviewed!” method, (she now has a technical thread to filter such antics) to the more reasoned technical response from NCAR’s Piers Forster, Jonathan Gregory & Ed Hawkins in Comments on the GWPF climate sensitivity report, and subsequently botched reaction by Sherwood in the Guardian, I thought it might be useful to point out another paper on the evaluation of climate models.
While this paper doesn’t address the issue of climate sensitivity, but focuses more on quantifying and rating divergence, it does have one thing in common with Lewis and Crok; it compares climate model output to observational data.
Though, not wholly directly; it uses the ERA40 reanalysis data (The ERA40 reanalysis data set from ECMWF, is the same reanalysis dataset deriving temperature near the north pole by the Danish Meteorological Institute) it does point out that there are significant divergences from model output and then scores them. I noted how far off GISS-AOM was in their figure 2 which shows hindcast performance from 1961-1990:
Some models scored well in hindcast, staying close to the ERA40 reanalysis data, so they aren’t all bad. The point is that models all give an estimate, and while we can argue that averages/ensembles represent the best guess for reality, the true measure of reality comes back to comparison of real-world data. What Lewis and Crok did is nothing more than evaluate many climate models for their performance against real world data:
In recent years it has become possible to make good empirical estimates of climate sensitivity from observational data such as temperature and ocean heat records. These estimates, published in leading scientific journals, point to climate sensitivity per doubling of CO2 most likely being under 2°C for long-term warming, with a best estimate of only 1.3-1.4°C for warming over a seventy year period.
I think much of the negative reaction to the report has to do with the “who and how” of the Lewis and Crok report production, rather than the content, and that is used as an excuse to ignore it rather than to show why it might be wrong. By saying it isn’t peer reviewed, it makes it easy to say (lazily) that the Lewis and Crok report has no value. However, as we’ve seen recently, where…
Publishers withdraw more than 120 gibberish papers, Conference proceedings removed from subscription databases after scientist reveals that they were computer-generated.
…peer review is no guarantee that one paper is necessarily better than another. The problem with peer review is that it relies on volunteerism, and I suspect many scientists asked to review are often too busy to give the level of commitment required to fully analyze, test, and/or replicate a paper’s data/methodology they are asked to look at.
In the case of Lewis and Crok, because of the high visibility, you can bet there will be many looking at it far more carefully, looking for flaws, than if it had three referees. Either the paper is right or it is wrong, I’m sure we’ll find out in the future, if not by challenges, but by Mother Nature being the arbiter of truth.
This paper is from The Society for Industrial and Applied Mathematics, SIAM:
Evaluating climate models
The simulation of elements affecting the Earth’s climate is usually carried out by coupled atmosphere-ocean circulation models, output from which is used to provide insights into future climate states. Climate models have traditionally been assessed by comparing summary statistics (such as global annual mean temperatures) or point estimates from simulated models to the corresponding observed quantities.
A paper published last December in the SIAM/ASA Journal on Uncertainty Quantification argues that it is more appropriate to compare the distribution of climate model output data (over time and space) to the corresponding distribution of observed data. Distance measures between probability distributions, also called divergence functions, can be used to make this comparison.
The authors evaluate 15 different climate models by comparing simulations of past climate to corresponding reanalysis data. Reanalysis datasets are created by inputting climate observations using a given climate model throughout the entire reanalysis period in order to reduce the effects of modeling changes on climate statistics. Historical weather observations are used to reconstruct atmospheric states on a global grid, thereby allowing direct comparison to climate model output.
View the paper:
Using Proper Divergence Functions to Evaluate Climate Models*
Abstract:
It has been argued persuasively that, in order to evaluate climate models, the probability distributions of model output need to be compared to the corresponding empirical distributions of observed data. Distance measures between probability distributions, also called divergence functions, can be used for this purpose. We contend that divergence functions ought to be proper, in the sense that acting on modelers’ true beliefs is an optimal strategy. The score divergences introduced in this paper derive from proper scoring rules and, thus, they are proper with the integrated quadratic distance and the Kullback–Leibler divergence being particularly attractive choices. Other commonly used divergences fail to be proper. In an illustration, we evaluate and rank simulations from 15 climate models for temperature extremes in a comparison to reanalysis data.
“While the reactions to Lewis and Crok rage from intellectually lazy trolling”
rage->range
REPLY: No, that’s accurate – A
Judging from the various splutterings, rage might be closer than range!
Peer review should be opponent review. Like old Timex watches, new research should be accepted only after it’s taken a lickin’ and kept on tickin’.
Gary says: March 6, 2014 at 8:18 am “Peer review should be opponent review. Like old Timex watches, new research should be accepted only after it’s taken a lickin’ and kept on tickin’.”
Well said. Falsification separates science from non-sense, the Problem of Demarcation.
The Peer Review trolls on Dr. Curry’s blog seem to have completely missed the latest revelation of the worth of “peer review” that WUWT and Joanne Nova highlighted. But given who they are, that is not surprising.
Wh isnt there a peer review journal “Journal of Audited Climatic Studies” where reports like this can be published and then reviewed by peers that agree with them just like on the warmer side?
Couldn’t this have been published in one of the PLOS-type online journals? Even now?
People forget that the IPCC uses non-peer reviewed literature which they call gray literature. So if anyone want to complain then start with the alleged ‘Gold Standard’ of Climastrology. 🙂
Having seen the absolute garbage for comments over at JC’s blog last night, I was so appalled I was compelled to leave a comment. I was used to seeing more technical comments on that site and view it as a valuable resource. The comments on this paper amounted to nothing but pure obstructionism and name calling from both sides. It was pathetic. I said that in very unflattering terms. JC, must have agreed. Good for her, for both creating a technical comment thread and for being enough of a scientist to actually consider all the papers available, no matter who they agree with politically. Science has always been so import to me since I was a kid, and the current state of science activism has been an embarrassment.
PLOS and that type are fee-for-publication. PLOS $1400 in 2010
Any report is going to get much more “review” here than it would being submitted anywhere else. If you have any cahones, you make your report totally public, since you are interested in the truth, right?
The arse on Ms. Curry’s blog is really, I mean, really tiring. Real denial, every day.
AR5 of the IPCC used non-peer reviewed sources.
Forgot the link
https://www.ipcc.ch/meetings/session32/inf04_p32_review_ipcc_proc_proced_notes_informal_task_group.pdf
william says: Why isn’t there a peer review journal “Journal of Audited Climatic Studies” where reports like this can be published and then reviewed by peers that agree with them just like on the warmer side?
I would like to see a journal that accepts debatable areas of climatology, and that strictly adjudicates the back and forth among the various points of view. — kind of like this website, but more formal —
I’m skeptical of any ECS figure much different from 1. The ice cores show no high frequency correlation between CO2 and T, and the paleo data show no correlation over the eons. We are left with millennial correlation which is easily explained as a tandem response to ice sheet extension. What’s left? One warm period from 1970-2000, and one flat period since. Not much to go on. –AGF
“Peer review should be opponent review. :
Indeed. I’m not a great writer, but I am a better writer than I would be if I haven’t been giving my copy to an old college classmate who has ruthlessly found every error I’ve ever made for the past 25 years. I likewise return the favour. Its a healthy respect, otherwise, but I don’t WANT someone nice marking my copy. I WANT perfection, or as close to it as I can get by deadline.
Doing it any other way just reminds me of my wife asking me if she looks good in something. The answer is ALWAYS yes…and I pay the price if the answer was supposed to be NO.
I thought it was insanely readable for a climate science paper. Highly irregular, if you ask me. 😉
Anthony,
While the folks over at Ed Hawkins page may be willing to concede a little bit on the climate sensitivity being lower than IPCC assessments, they quickly move to keep the dangerous climate change meme alive by viewing the RCP8.5 scenario (the most extreme scenario the IPCC considered when “sampling the full range of published scenarios” (AR5 Ch12)) as “business-as-usual.” This is an improper characterization of that scenario. BAU is not a static technology scenario, but rather a dynamic one (largely free from government interference).
-Chip
“…..While this paper doesn’t address the issue of climate sensitivity, but focuses more on quantifying and rating divergence, it does have one thing in common with Lewis and Crok; it compares climate model output to observational data…..”
This is simply OUTRAGEOUS !!! There is no serious scientist – living or dead – that would ever consider even for a nano-second actually checking the results of a model with REAL DATA .
Why, that would be akin – very roughly speaking – to conducting controlled experiments to validate a theory . Or far, far worse, even bothering to conduct an experiment at all!!
What is this world coming to .
It has been PROVEN, REPEATEDLY, that real science is based on opinion and consensus and the educated, sophisticated science movies starring AL GORE and the wise words of assorted railroad engineers.
I intend to immediately cancel my subscription to the National Enquirer, and will only re-up when they once again write real science articles.
Peer review; non-sense. Get people in there to replicate “shtuff” and see what happens. Reality wins. Every time, bring on reality.
Science is like a great big neverending court trial, in which proponents of a theory put forth their arguments and data, and theory opponents provide contrary arguments and data. The scientific community acts as judge and jury. Like any man-made system, the results are not always
correct, since a lot depends upon the abilities of the participants. But no scientific theory or law can ever be “proven” to be anything more than a useful tool for getting along with the environment. Strictly speaking, a “settled science” is a logical impossibility, since there is no conceivable way in which one can ever prove that a reigning theory is the only possible explanation for the events with which it deals, no matter how accurately (remember Newtonian laws, long considered perfect?).
Here is the Equilibrium Climate Sensitivity over the past 15 million years. There are 1790 individual estimates in this chart.
http://s28.postimg.org/nxyzh50zx/Equil_CO2_Sensitivity_40range_Last_15_Mys.png
What I have done here is use all the CO2 estimates available from reliable methods within the timeline and backfit the highest resolution temperature history estimates to the exact time that the CO2 estimates came from. So the time is exactly the same for CO2 and temperature.
(This is only showing the numbers from -40.0C per doubling to +40.0C per doubling but there are numerous numbers outside this range as well).
Random comes to mind and/or Albedo is really controlling the temperatures, not CO2.
Chip Knappenberger: BAU is not a static technology scenario, but rather a dynamic one (largely free from government interference).
Thank you!
Maybe many people have written that, but in my readings I have not seen it from anyone but me. That is an idea that needs to be promoted more.
From the report –
>>> It is almost universally accepted that by itself the equilibrium warming effect
of a doubling of the carbon dioxide concentration is slightly more than 1◦C.
Why then do models have an average ECS of 3◦C? This is due to the knock-on
effects of the initial warming: so-called ‘positive feedbacks’. Combined with
the initial warming, three feedbacks – water vapour, lapse rate and albedo –
can together explain an ECS value of 2◦C. The remaining 1◦C required to explain
theGCMaverage of 3◦C is accounted for primarily by cloud feedback and
related adjustments.
Conclusion
Climate models predict upper atmosphere moistening which triples the greenhouse effect from man-made carbon dioxide emissions. The new satellite data from the NASA water vapor project shows declining upper atmosphere water vapor during the period 1988 to 2001. It is the best available data for water vapor because it has global coverage. Calculations by a line-by-line radiative code show that upper atmosphere water vapor changes at 500 mb to 300 mb have 29 times greater effect on OLR and temperatures than the same change near the surface. The cooling effect of the water vapor changes on OLR is 16 times greater than the warming effect of CO2 during the 1990 to 2001 period. Radiosonde data shows that upper atmosphere water vapor declines with warming. The IPCC dismisses the radiosonde data as the decline is inconsistent with theory. During the 1990 to 2001 period, upper atmosphere water vapor from satellite data declines more than that from radiosonde data, so there is no reason to dismiss the radiosonde data. Changes in water vapor are linked to temperature trends in the upper atmosphere. Both satellite data and radiosonde data confirm the absence of any tropical upper atmosphere temperature amplification, contrary to IPCC theory. Four independent data sets demonstrate that the IPCC theory is wrong. CO2 does not cause significant global warming.
http://wattsupwiththat.com/2013/03/06/nasa-satellite-data-shows-a-decline-in-water-vapor/
Why didn’t they examine water vapor and the false sign on the feedback?
What, precisely, is it that makes a climate model a climate model, and why would anyone believe that the output from distinct models represents samples from the same population — which I understand to be an important element in looking at the statistical properties of their output, unless the population is defined so loosely as to be relatively meaningless?
My work requires me to construct numerical models. If I waved my magic wand and said the output of one of them represented “global temperature anomaly”, and say its output, appropriately scaled, matched some portion of the “historical” historical temperature record, would that make it a “climate model”? Or does the state of mind of the “expert” developing the model matter?
Hasn’t Mr. Eschenbach demonstrated that most “climate model” output is fairly well represented by a simple auto-regressive model, or is that misrepresenting his work?
Bill Illis says:
March 6, 2014 at 11:09 am
===============================
Very nice. –AGF
|| Gary says: March 6, 2014 at 8:18 am “Peer review should be opponent review. Like old Timex watches, new research should be accepted only after it’s taken a lickin’ and kept on tickin’.”
| Well said. Falsification separates science from non-sense, the Problem of Demarcation.
— As opposed to the intellectually bankrupt “pal review” that (sfaict) every junk-science paper from the hockey team underwent….
Science does self-correct, eventually, and the chickens look to be coming in to roost. To the vocal dismay of the political scientists and their allies.
Peter D. Tillman
Professional geologist, amateur climatologist
There has been one for a bit over a year–the Dutch govt’s Climate Dialog site. But it only holds confabs every three months (?) or so.
In Welsh language orthography the Welsh word “siop” is pronounced identically to the English “shop”. I wonder how SIAM should be pronounced.