After spending a year trying to get the data from the author without success, Nic Lewis has sent a letter to the editor of Geophysical Research Letters (GRL) and has written to me to ask that I bring attention to his letter published at Judith Curry’s website, and I am happy to do so. He writes:
I would much appreciate it if you could post a link at WUWT to an article of mine (as attached) that has just been published at Climate Etc. It concerns the alteration of data used in an important climate sensitivity study, Forest 2006, with a radical effect on the resulting climate sensitivity estimated PDF.
I’m including the foreword here (bolding mine) and there is a link to the entire letter to the editor of GRL.
Questioning the Forest et al. (2006) sensitivity study
By Nicholas Lewis
Re: Data inconsistencies in Forest, Stone and Sokolov (2006) GRL paper 2005GL023977 ‘Estimated PDFs of climate system properties including natural and anthropogenic forcings‘
In recent years one of the most important methods of estimating probability distributions for key properties of the climate system has been comparison of observations with multiple model simulations, run at varying settings for climate parameters. Usually such studies are formulated in Bayesian terms and involve ‘optimal fingerprints’. In particular, equilibrium climate sensitivity (S), effective vertical deep ocean diffusivity (Kv) and total aerosol forcing (Faer) have been estimated in this way. Although such methods estimate climate system properties indirectly, the models concerned, unlike AOGCMs, have adjustable parameters controlling those properties that, at least in principle, are calibrated in terms of those properties and which enable the entire parameter space to be explored.
In the IPCC’s Fourth Assessment Report (AR4), an appendix to WGI Chapter 9, ‘Understanding and attributing climate change’[i], was devoted to these methods, which provided six of the chapter’s eight estimated probability density functions (PDFs) for S inferred from observed changes in climate. Estimates of climate properties derived from those studies have been widely cited and used as an input to other climate science work. The PDFs for S were set out in Figure 9.20 of AR4 WG1, reproduced below.
The results of Forest 2006 and its predecessor study Forest 2002 are particularly important since, unlike all other studies utilising model simulations, they were based on direct comparisons thereof with a wide range of instrumental data observations – surface, upper air and deep-ocean temperature changes – and they provided simultaneous estimates for Kv and Faer as well as S. Jointly estimating Kv and Faer together with S is important, as it avoids dependence on existing very uncertain estimates of those parameters. Reflecting their importance, the IPCC featured both Forest studies in Figure 9.20. The Forest 2006 PDF has a strong peak which is in line with the IPCC’s central estimate of S = 3, but the PDF is poorly constrained at high S.
I have been trying for over a year, without success, to obtain from Dr Forest the data used in Forest 2006. However, I have been able to obtain without any difficulty the data used in two related studies that were stated to be based on the Forest 2006 data. It appears that Dr Forest only provided pre-processed data for use in those studies, which is understandable as the raw model dataset is very large.
Unfortunately, Dr Forest reports that the raw model data is now lost. Worse, the sets of pre-processed model data that he provided for use in the two related studies, while both apparently deriving from the same set of model simulation runs, were very different. One dataset appears to correspond to what was actually used in Forest 2006, although I have only been able to approximate the Forest 2006 results using it. In the absence of computer code and related ancillary data, replication of the Forest 2006 results is problematical. However, that dataset is compatible, when using the surface, upper air and deep-ocean data in combination, with a central estimate for climate sensitivity close to S = 3, in line with the Forest 2006 results.
The other set of data, however, supports a central estimate of S = 1, with a well constrained PDF.
I have written the below letter to the editor-in-chief of the journal in which Forest 2006 was published, seeking his assistance in resolving this mystery. Until and unless Dr Forest demonstrates that the model data used in Forest 2006 was correctly processed from the raw model simulation run data, I cannot see that much confidence can be placed in the validity of the Forest 2006 results. The difficulty is that, with the raw model data lost, there is no simple way of proving which version of the processed model data, if either, is correct. However, so far as I can see, the evidence points to the CSF 2005 version of the key surface temperature model data, at least, being the correct one. If I am right, then correct processing of the data used in Forest 2006 would lead to the conclusion that equilibrium climate sensitivity (to a doubling of CO2 in the atmosphere) is close to 1°C, not 3°C, implying that likely future warming has been grossly overestimated by the IPCC.
This sad state of affairs would not have arisen if Dr Forest had been required to place all the data and computer code used for the study in a public archive at the time of publication. Imposition by journals of such a requirement, and its enforcement, is in my view an important step in restoring trust in climate science amongst people who base their beliefs on empirical, verifiable, evidence.
Nic Lewis
==============================================================
Just let me say that there’s movement afoot to address the issues brought up about reproducibility in journal publications in the last paragraph. I’ll have more on this at a future date.
Here’s the foreword and letter to the GRL editor in PDF form: Post on Forest 2006 GRL letter final
This figure from that letter by Lewis suggests a lower climate sensitivity to a doubling of CO2 than the original:
-Anthony
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Pamela Gray says:
June 25, 2012 at 10:40 am
Once again I don’t get it with the lost data…..
___________________________
I am with Pamela on this. I did not even do a Master’s thesis only a couple of Senior Year Topics papers, one in geology and one in Chemistry. I still have all the information and even gave a copy of the Geo research to a geologist who asked me for the information at a National Speleological Convention of all places.
If you can not come up with the raw data and have it validated and verified by others it is not science PERIOD! Think cold fusion and the more recent CERN finds faster than light particles? and the update CERN researchers find flaw in faster-than-light measurement.
I wonder if it would be reasonable to try and reproduce the top 15 climate paper, in terms of cites?
surely this should be a sticky top post?
timetochooseagain: ‘ the worst offender appears to be “Knutti 02″ ‘
Indeed – not a very useful estimate of sensitivity IMO, since it reflects and unweighted average of 5 different ocean models, and uses a weak yes/no rejection test for the simulation-observation mismatch. The Knutti paper didn’t actually include either a sensitivity PDF or any distribution from which one could be calculated, except for a 5 bar histogram (2 K per bar) in the SI. I presume that the IPCC ran this through some sort of smoothing algorithm.
Silly old me thinking science was about reproducability from the data. Now if you can’t get the data then it’s not science in my opinion.
Here is the great irony of the “Lost Data”/Rcord Keeping mystery.
It seems that while CRU/IPCC scientists can get away with “The dog ate my home work” The United Nations (OIE) 2005 Draft Guide To Good Farming Practices wanted to hold farmers around the world to much higher standards.
If the UN expects FARMERS to keep and have available those type of very specific records, how come PhD scientists are not held to the same standard as a lowly farmer? Especially when the scientist will have a much larger effect than any one farmer – Double Standards anyone?
In light of the above UN document, the “dog ate my homework” excuse can not be justified. The IPCC should be throwing out ALL scientific work that is not as precisely documented as the UN was demanding of farmers who are NOT even receiving public research grants to cover the costs.
Nic Lewis says:
June 25, 2012 at 3:24 pm
timetochooseagain: ‘ the worst offender appears to be “Knutti 02″ ‘
Indeed – not a very useful estimate of sensitivity IMO, since it reflects and unweighted average of 5 different ocean models, and uses a weak yes/no rejection test for the simulation-observation mismatch.
———————————————————-
Such papers obviously do not harm an author’s career in this branch of science. Co-author Thomas A Stocker is now co-chair of the coming IPCC report.
Lost the data? Good enough for “climate science”.
Don’t blame Dr Forest. I’m sure he had funds set aside to pay a grad student to do some filing, but his boss insisted that the money be spent instead on
a junket to an exotic location a well earned holidaya boring conference in some godforsaken third world hellhole.Unfortunately, Dr Forest reports that the raw model data is now lost.
===================
Can’t prove scientific fraud if the data is lost. Is simply means that Forest et al. (2006) is scientific nonsense. It cannot be reproduced, even by the author. It has the same value as used toilet paper. It is paper, but it is covered in crap.
No replicability ☰ No science.
The paper must be withdrawn, invalidating all references to it.
Why should Dr. Forest make his data available to Nic when Nic’s aim is to find something wrong with it? /sarc
How to make Money in Climate Science
1. find a major unanswered question.
3. question top scientists to see what they will accept as an answer
3. cherry pick data and methods to arrive at that answer
4. re-label this technique “training” – it makes it sound intelligent.
5. publish the result.
The results will seem correct to fellow scientists, especially those at the top, so they wont bother to check the math. Everyone will be impressed you have answered the hard question. More so because you will have proven their best guess correct and made them look good in the process. You will advance in your career in science. Fame and fortune will follow.
If anyone does question the results:
6. lose the data and methods
Lucy Skywalker says: June 25, 2012 at 1:33 pm Quoting from the Royal Society, “should a paper be withdrawn from a journal, after it has already been heavily cited?..”
Lucy, should a mineral discovery be withdrawn from a prospectus after it had been found that the assays were badly wrong, or lost, or both?” People can go to prison for this.
Is it not self-evident that there should be an active, continuing process of demotion into separate storage of unacceptable papers and correction of acceptable papers when, for example, better data become available, like new temperature series used in proxy calibration? Although Science is seldom expressed in absolutes, should not there be a classification like G for general exhibition, M+ for mature audiences, R for dubious and X for fail? Why, there could even be a ratings journal that logs papers and shows event that overtake them.
Have addendum, corregidum, erratum, etc been wiped from the repertoires of authors?
There is evidence for a quite bad structural/organisational failure in the halls of academia when the emphasis is on floating new ideas, with an almost 100% failure to wipe the slate clean of the baddies. As noted ad nauseum, much of this starts with the gross and frequent failure of peer review.
The paper in question, I believe, is Forest, Stone and Sokolov (2006)
“Estimated PDF’s of climate system properties including natural and anthropogenic forcings”
Yesterday, out of interest, I googled ‘Dr C E Forest PSU’ to find out more about the man.
Unless I am mistaken, no such paper now exists in his bio of ‘Selected Publications’, which bearing in mind it’s obvious importance seems a little odd to say the least.
Nic Lewis,
Congratulations on your endeavour and clearly sustained work focus. Thank you also for the article and subsequent explanations. Most interesting, I am still working my way through reading this post.
However. I still have a problem understanding the term (and focus) on sensitivity. I was taught about sensitivity and specificity. Type I & II errors.
I do not understand why there continues to be a focus on sensitivity when, to my understanding, the question should be on the specificity. .. an incorrectly formulated hypothesis.
Would you have an explanation to this question please?
Nic Lewis,
Thank you for responding here. While most commenters are most concerned about the the fact the data is “lost” and the code withheld, I am more interested in the policy implications of a possible reduction in the climate sensitivity (central estimate and ‘fat tail’). Therefore, could you please advise what has happened in response to your finding, reported on Judith Curry’s site in July 2006, that the IPCC AR4 had replotted (apparently incorrectly and without fully explaining why) the climate sensitivty results of Forster and Gregory (2006)?
For the benefit of other readers I’ll provide some more background, links and more questions below.
In July 2011, you suggested that IPCC had wrongly replotted the Forster/Gregory06 paper http://judithcurry.com/2011/07/05/the-ipccs-alteration-of-forster-gregorys-model-independent-climate-sensitivity-results/ . IPCC’s replot moved the median climate sensitivity from about 1.6C to 2.3C and gave a much fatter tail.
In a comment upthread (@ur momisugly June 25, 2012 at 10:56 am) you said:
Is that generally accepted? (this is an innocent question; I have no idea what has happened since with that). Does your comment mean that it is now generally accepted that the IPCC replot of Forster/Gregory06 was wrong? Does this mean that the now accepted interpretation of that paper is that it suggests the median climate sensitivity is 1.6C? What have the autjhors, IPCC and the journal that published the paper said and done?
Could you please elaborate (for a non specialist) on the significance of this.
Hm, correct me if I am wrong, but doesn’t much of the work that finds that eventually their could be net economic harm from AGW depend crucially on the PDF of climate sensitivity-and require the fat tails for their conclusions? Such studies already find that the “ideal” policy is very close to doing nothing, even under the assumption of many negative “impacts” which, frankly, do not exist. So without the fat tail, Nordhaus can kiss his Carbon Tax (already small) goodbye, no?
Andrew,
Yes. Nordhaus conclusions are headed: “Not so dismal conclusions” (no evidence of strong tail dominance)
Nordhaus (2012) Economic policy in the face of sever tail events
http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9779.2011.01544.x/full
I hate acknowledging my ignorance, but honesty compels me to.
I can’t make head nor tail of these graphs.
What are the abbreviations, why do they have the axes they do, what do you think they prove and why do you think so?
I’ve an IQ that’s acceptable, a physics and chemistry education to year 10, Advanced Math for year 11, some statistics at tertiary level, a lifetime science enthusiasm including most of Sagan, Asimov, Gould et al’s popularised science- in short, I’m vastly better educated than the average member of the population(*)- and I still have no clear conception of what these graphs are supposed to prove or why you think they do so. My point being that if I can’t follow them, neither can many others of your readers. Can the OP or commentators please help me?
(*) I said I was honest, not humble 😉
Geoff Sherrington says:
June 26, 2012 at 4:16 am
…..Is it not self-evident that there should be an active, continuing process of demotion into separate storage of unacceptable papers and correction of acceptable papers when, for example, better data become available, like new temperature series used in proxy calibration? Although Science is seldom expressed in absolutes, should not there be a classification like G for general exhibition, M+ for mature audiences, R for dubious and X for fail? Why, there could even be a ratings journal that logs papers and shows event that overtake them…..
_______________________________________
Yes it is sel-evident and it has already started with Retraction Watch
The latest headline is “Following investigation, Erasmus social psychology professor retracts two studies, resigns”
WOW, could not produce the raw data… cherry-picked the data…. statistically highly unlikely…. GEE where have we heard that before?
Other branches of science are starting to clean up their act, isn’t it about time Climatology was held to the same classic science standard?
Kaboom says:
June 25, 2012 at 10:36 am
I’d feel for those who used Forest’s work as a key ingredient for their own studies as this pretty much invalidates anything they’ve come up with. But alas they’re likely to be cut from the same cloth of preconcept-dictates-research academics and thus have not added to the body of science with their papers anyway.
=======================
The last thing in the world that the 100+ scientists that have used F& al in their citation will want is for the paper to be withdrawn. This would call into question their own papers. They have a very strong vested interest is supporting F & al regardless of quality.
Would it not also make sense to file a FOIA request for the raw data, etc. to the granting agencies and academic institutions involved?
/Mr Lynn
Peter Lang
I have answered your question, and supplementary question, on the thread at Climate Etc
Leo Morgan
May I suggest that you read Chapter 9 of the IPCC AR4 WG1 report, from which the first graph comes, and the WG1 Glossary? Available at http://www.ipcc.ch/publications_and_data/ar4/wg1/en/contents.html
Climate sensitivity is defined as the rise in global temperature resulting from a doubling of carbon dioxide concentration in the atmosphere, after the climate system reaches a new equilibrium, with other external factors being held constant, roughly speaking.
Remember that the leaps in theoretical physics by Einstein were accompanied quite passionately by the call to verify them with empirical testing. Today the method is hide, profit, and move on before anyone bothers to check. But then who would want to check if the consequences amount to professional tire slashings from the new blacklist enforcers of the climate change ecosystem?