After spending a year trying to get the data from the author without success, Nic Lewis has sent a letter to the editor of Geophysical Research Letters (GRL) and has written to me to ask that I bring attention to his letter published at Judith Curry’s website, and I am happy to do so. He writes:
I would much appreciate it if you could post a link at WUWT to an article of mine (as attached) that has just been published at Climate Etc. It concerns the alteration of data used in an important climate sensitivity study, Forest 2006, with a radical effect on the resulting climate sensitivity estimated PDF.
I’m including the foreword here (bolding mine) and there is a link to the entire letter to the editor of GRL.
Questioning the Forest et al. (2006) sensitivity study
By Nicholas Lewis
Re: Data inconsistencies in Forest, Stone and Sokolov (2006) GRL paper 2005GL023977 ‘Estimated PDFs of climate system properties including natural and anthropogenic forcings‘
In recent years one of the most important methods of estimating probability distributions for key properties of the climate system has been comparison of observations with multiple model simulations, run at varying settings for climate parameters. Usually such studies are formulated in Bayesian terms and involve ‘optimal fingerprints’. In particular, equilibrium climate sensitivity (S), effective vertical deep ocean diffusivity (Kv) and total aerosol forcing (Faer) have been estimated in this way. Although such methods estimate climate system properties indirectly, the models concerned, unlike AOGCMs, have adjustable parameters controlling those properties that, at least in principle, are calibrated in terms of those properties and which enable the entire parameter space to be explored.
In the IPCC’s Fourth Assessment Report (AR4), an appendix to WGI Chapter 9, ‘Understanding and attributing climate change’[i], was devoted to these methods, which provided six of the chapter’s eight estimated probability density functions (PDFs) for S inferred from observed changes in climate. Estimates of climate properties derived from those studies have been widely cited and used as an input to other climate science work. The PDFs for S were set out in Figure 9.20 of AR4 WG1, reproduced below.
The results of Forest 2006 and its predecessor study Forest 2002 are particularly important since, unlike all other studies utilising model simulations, they were based on direct comparisons thereof with a wide range of instrumental data observations – surface, upper air and deep-ocean temperature changes – and they provided simultaneous estimates for Kv and Faer as well as S. Jointly estimating Kv and Faer together with S is important, as it avoids dependence on existing very uncertain estimates of those parameters. Reflecting their importance, the IPCC featured both Forest studies in Figure 9.20. The Forest 2006 PDF has a strong peak which is in line with the IPCC’s central estimate of S = 3, but the PDF is poorly constrained at high S.
I have been trying for over a year, without success, to obtain from Dr Forest the data used in Forest 2006. However, I have been able to obtain without any difficulty the data used in two related studies that were stated to be based on the Forest 2006 data. It appears that Dr Forest only provided pre-processed data for use in those studies, which is understandable as the raw model dataset is very large.
Unfortunately, Dr Forest reports that the raw model data is now lost. Worse, the sets of pre-processed model data that he provided for use in the two related studies, while both apparently deriving from the same set of model simulation runs, were very different. One dataset appears to correspond to what was actually used in Forest 2006, although I have only been able to approximate the Forest 2006 results using it. In the absence of computer code and related ancillary data, replication of the Forest 2006 results is problematical. However, that dataset is compatible, when using the surface, upper air and deep-ocean data in combination, with a central estimate for climate sensitivity close to S = 3, in line with the Forest 2006 results.
The other set of data, however, supports a central estimate of S = 1, with a well constrained PDF.
I have written the below letter to the editor-in-chief of the journal in which Forest 2006 was published, seeking his assistance in resolving this mystery. Until and unless Dr Forest demonstrates that the model data used in Forest 2006 was correctly processed from the raw model simulation run data, I cannot see that much confidence can be placed in the validity of the Forest 2006 results. The difficulty is that, with the raw model data lost, there is no simple way of proving which version of the processed model data, if either, is correct. However, so far as I can see, the evidence points to the CSF 2005 version of the key surface temperature model data, at least, being the correct one. If I am right, then correct processing of the data used in Forest 2006 would lead to the conclusion that equilibrium climate sensitivity (to a doubling of CO2 in the atmosphere) is close to 1°C, not 3°C, implying that likely future warming has been grossly overestimated by the IPCC.
This sad state of affairs would not have arisen if Dr Forest had been required to place all the data and computer code used for the study in a public archive at the time of publication. Imposition by journals of such a requirement, and its enforcement, is in my view an important step in restoring trust in climate science amongst people who base their beliefs on empirical, verifiable, evidence.
Nic Lewis
==============================================================
Just let me say that there’s movement afoot to address the issues brought up about reproducibility in journal publications in the last paragraph. I’ll have more on this at a future date.
Here’s the foreword and letter to the GRL editor in PDF form: Post on Forest 2006 GRL letter final
This figure from that letter by Lewis suggests a lower climate sensitivity to a doubling of CO2 than the original:
-Anthony
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
I’ve been doing my best to help this along. This should be brought to the attention of Tom Hammond on the House Science Committee. We had a long discussion about exactly this sort of thing, and since the US government almost always pays or helps pay for the work, it isn’t crazy to insist on it.
rgb
The Journal of Irreprocible Results was always one of my favorites!
Does this mean GRL paper 2005GL023977 should now be considered “grey literature”?
Make that Irreproducible. It still exists!
http://www.jir.com/
You can’t show the data means you don’t have a paper.
As if there was such a thing as global climate sensitivity,
calculated by a mere spatial average of local sensitivities,
but valid for predicting ‘global average temperature’ (whatever that is)
under future emission scenarios.
Talk about far-fetched.
Worse yet, they average various bogus ‘sensitivites’ to get a ‘likely’ sensitivity.
This garbage is as totally removed from climatic reality
as Keynsian economics is from economic reality.
Hehe, looks like the high sensitivity “fat tail” is a phantom.
Keep in mind that the scariest climate scenarios are dependent on the “fat tail” for their plausibility, that is, they require a non-negligible probability of sensitivities greater than 4 K per doubling of CO2.
I no longer bother to read or try to understand the information published in these journals as you simply can;t believe anything they tell you anymore. I make it a point to tell this to everyone I know and as it is known that I’m always interested in the science end of things , people I know are also doubtful of things science published in the msm. So I do get some revenge on the liers afterall. But it is a sad and bad time for scientific advancement now.
Kaboom
You can’t show the data means you don’t have a paper.
… as a last resort – the paper challenged use the inside of the toilet roll!
“Dr Forest reports that the raw model data is now lost.”
The dog ate my homework defense! – A classic ‘Team’ response!:
http://rogerpielkejr.blogspot.com/2009/08/we-lost-original-data.html
Dr. Forest is obviously a ‘Team’ player!
Quite so.
“Unfortunately, Dr Forest reports that the raw model data is now lost.”
Makes me wonder if the “dog ate my homework” excuse worked for Forest in grammar school, or if it now comes down to “…the accumulation of the raw model data is left as an exercise for the reader.”.
Lost in the Forest?
If I understand the IPCC’s chart, the “lines” at the bottom are meant to be the PDFs collapsed into central estimates with confidence intervals. Notice how the central estimates are pretty much all to the right (higher sensitivity) than the peaks of the PDFs. It seems to me that they may have used inappropriate methods for determining central estimates, given “fat tailed” (ie skewed) distributions.
I’d feel for those who used Forest’s work as a key ingredient for their own studies as this pretty much invalidates anything they’ve come up with. But alas they’re likely to be cut from the same cloth of preconcept-dictates-research academics and thus have not added to the body of science with their papers anyway.
Once again I don’t get it with the lost data. I am just a poedunk nothin in terms of research and have only a decade’s old master’s level research endeavor archived at Oregon State University plus an article of that research in a major journal. I have no Ph.D. attached to my name, just a bachelor and two masters degrees and my resume does not come with a vitae. Yet I still have my raw data. I have kept it to this day. I still have a drawing of the electrical components used to generate the stimulus I used. I still have a poloroid of the stimulus captured on a spectral analyser. I still have the original typed on a Wang computer masters article that was then copied into the archive volume at OSU. And I no longer practice in that field.
Did this guy skip research 101 class?
What happens in the academic world if you accuse someone of altering data?
I note in passing that Cris Forest is now a colleague of Michael Mann at Penn State. Maybe they can undertake a joint project on data management.
timetochooseagain:
The filled circles on the 5-95% ranges (which aren’t true confidence intervals) in the bottom section of the IPCC figure are the medians, which as you say are to the right of the peaks of the PDFs. That is actually to be expected, because errors in estimating changes in forcings and ocean heat uptake greatly exceed errors in temperature data. But almost all the distributions are more skewed than that effect would account for: their tails are indeed too fat. In the case of Forster/Gregory 06, that is because the IPCC changed the distribution onto an incorrect basis. And the very bumpy and other strange shaped distributions are self-evidently flawed.
Losing your data is simply the most extreme version of altering your data.
“Team Dictionary”
Lost- under no circumstances release raw data to anyone not on the team. All they want to do is find something wrong with it.
And real science takes another hit from the team.
What, no trolls?
Reproducibility? Every time we run the same model (ie, computer program) with the same starting points and same data, we reproduce the same results.
/sarc
At least climategate didn’t show anyone losing raw data eh!
If one of Dr Forests students lost their coursework data would it be a fail?
It just amazes me that “scientists” don’t use any source controll programs, try that in any software company nowdays and you would be viewed as a clueless n00b.