An excerpt from Steve’s post at Climate Audit
Schneider replied that he had been editor of Climatic Change for 28 years and, during that time, nobody had ever requested supporting data, let alone source code, and he therefore required a policy from his editorial board approving his requesting such information from an author. He observed that he would not be able to get reviewers if they were required to examine supporting data and source code. I replied that I was not suggesting that he make that a condition of all reviews, but that I wished to examine such supporting information as part of my review, was willing to do so in my specific case (and wanted to do so under the circumstances) and asked him to seek approval from his editorial board if that was required.
This episode became an important component of Climategate emails in the first half of 2004. As it turned out (though it was not a point that I thought about at the time), both Phil Jones and Ben Santer were on the editorial board of Climatic Change. Some members of the editorial board (e.g. Pfister) thought that it would be a good idea to require Mann to provide supporting code as well as data. But both Jones and Santer lobbied hard and prevailed on code, but not data. They defeated any requirement that Mann supply source code, but Schneider did adopt a policy requiring authors to supply supporting data.
I therefore re-iterated my request as a reviewer for supporting data – including the residuals that Climategate letters show that Mann had supplied to CRU (described as his “dirty laundry”). The requested supporting data was not supplied by Mann and his coauthors and I accordingly submitted a review to Climatic Change, observing that Mann et al had flouted the new policy on providing supporting data. The submission was not published. I observed on another occasion that Jones and Mann (2004) contained a statement slagging us, based on a check-kiting citation to this rejected article.
During this exchange, I attempted to write thoughtfully to Schneider about processes of due diligence, drawing on my own experience and on Ross’ experience in econometrics. The correspondence was fairly lengthy; Schneider’s responses were chatty and cordial and he seemed fairly engaged, though the Climategate emails of the period perhaps cast a slightly different light on events.
Following the establishment of a data policy at Climatic Change, I requested data from Gordon Jacoby – which led to the “few good men” explanation of non-archiving (see CA in early 2005) and from Lonnie Thompson (leading to the first archiving of any information from Dunde, Guliya and Dasuopu, if only summary 10-year data inconsistent with other versions.) Here Schneider accomplished something that almost no one else has been able to do – get data from Lonnie Thompson, something that, in itself, shows Schneider’s stature in the field.
It was very disappointing to read Schneider’s description of these fairly genial exchanges in his book last year. Schneider stated:
The National Science Foundation has asserted that scientists are not required to present their personal computer codes to peer reviewers and critics, recognizing how much that would inhibit scientific practice.
A serial abuser of legalistic attacks was Stephen McIntyre a statistician who had worked in Canada for a mining company. I had had a similar experience with McIntyre when he demanded that Michael Mann and colleagues publish all their computer codes for peer-reviewed papers previously published in Climatic Change. The journal’s editorial board supported the view that the replication efforts do not extend to personal computer codes with all their undocumented subroutines. It’s an intellectual property issue as well as a major drain on scientists’ productivity, an opinion with which the National Science Foundation concurred, as mentioned.
This was untrue in important particulars and a very unfair account of our 2004 exchange. At the time, Schneider did not express any hint that the exchange was unreasonable. Indeed, the exchange had the positive outcome of Climatic Change adopting data archiving policies for the first time.
…
As I noted above, at his best, Schneider was engaging and cheerful – qualities that I prefer to remember him by. I was unaware of his personal battles or that he ironically described himself as “The Patient from Hell” – a title that seems an honorable one.
Read more at Climate Audit
hmm. I am just a one-article has been researcher but I still have my raw data on a disk, still have the computer program used to do the statistical analysis, still have the schematics of the electronic equipment used, still have hard copies of the raw tracings, still have drafts of the final article. If someone were to ask for these items, I would provide certified copies right now. The fact that others have duplicated our work, and at the same facility as well as at competing labs, is a feather in my cap, not an opportunity to horde so that we can continue to claim king of the hill.
Sphaerica, I’m thinking that arrogance can be seen in the mirror.
The exact opposite is true. Nobody has shown 20th century warming to be unusual or unprecedented.
@sphaerica
A computer model run is not an experiment. Computer models do not output data or facts. The code used to build these models is part of the process.
sphaerica says:
July 21, 2010 at 5:59 am
A major failing of every bystander in the issue of climate change is to apply their own personal, limited experiences and training to the problem, and then to believe that they, over everyone else, understand it best. There are far too many intelligent but ill-equipped engineers posting with hubris their confident (and wrong) assessment of climate science.
Note to world: study, learn, keep an open mind, and try to realize that arrogance and unjustified self confidence are failings, not virtues.
Do you believe that should apply to climate ‘scientists’ too? Is it not possible that these groupthinking cliques have arrived at a “wrong assessment of climate science” through their mutual pal review system?
Computer code is deterministic. Run the same code with the same data and the results are always the same.
The point is not whether somebody can replicate the results of somebody else’s computer model. Computer model runs are not experiments. The point is whether the computer model properly performs as claimed. Are the algorithms expressed properly? What values were assumed for the various forcings? And yes, is it full of bugs?
None of these things tell us if the model accurately models the real world. Only comparing model results to actual observation will do that. A computer model is an expression or illustration of an hypothesis.
Andrew30 says: July 21, 2010 at 12:56 am: – He is right.
Computer code is a language that trained people understand. I am a software engineer, and I know. I never hesistate to hand over code when a colleague asks me, I know he will understand. ( I do not write many comments, on principle. The code itself is the best comment).
Sometimes I need to be paid for the code (for my work!), but I don’t think that applies in Mann’s case.
Suppose there is a fragment in the paper written in, say, Spanish. Could Mann refrain from divulging this piece stating “it’s not documented, you would not understand it”? If it’s part of the paper or it’s supporting calculations, it must be divulged.
There is no excuse for not posting the code, except the desire to prevent replication. That NAS agreed to this is apalling.
tallbloke says: @sphaerica
July 21, 2010 at 7:36 am
“Do you believe that should apply to climate ‘scientists’ too? Is it not possible that these group-thinking cliques have arrived at a “wrong assessment of climate science” through their mutual pal review system?”
Ah, Tallbloke, you seem to have forgotten that Sphaerica is talking about the faith based cargo cult ‘science’ of CAGW, where the usual rules of Popperian falsification do not apply. The cosy and lucrative CAGW cabal can only survive via cronyism and deliberate obfuscation of the truth. It about as scientific as reading tea leaves!
It’s an easy mistake to make:-)
Just take my word for it you do not need Philosophiæ Naturalis Principia Mathematica to be published or understood. Somehow the concept of proof of an assertion is of no interest in ‘climate science’, Newton would not have enjoyed these times.
If by “idiosyncratic and uncommented” you mean rushed, amateur, and buggy then I’ll agree. But that’s the whole point. Raw data, if transformed by software, should be accompanied by at least the executable form of the software. No one’s sloppy source code is exposed there but it makes it much easier to test its functionality. Why make replication harder unless it’s because the author fears there will be flaws found? That doesn’t display much respect for the scientific method.
Scientists are supposed to encourage others in any reasonable way they can in replication. We’re not talking about the author buying beakers and reactants for others to use in replication here. Softare and source codes thereof are virtually free of time and cost to distribute. Put a copy on an FTP server and it’s done. Do you not support the scientific method, Toby, and reasonable efforts to assist others in replication?
I’m not a climatologist but as a programmer with decades of experience in commercial software development I must point out that a climatologist has as much expertise in writing software as I do in global circulation modeling. I understand quite well why a climatologist wouldn’t want a software expert examining his program source code. I think you know why too, Toby.
That said, there should be no question of releasing source code that was developed with public funding unless there is some national security reason for withholding it. That comes under the rubric of Freedom of Information law. The public has a legal right to it. Code developed with private funds of course can be released or not as the author wishes but it should least be viewed as not being fully supportive of the scientific method and cause for suspicion that the work may have deliberately hidden flaws.
The posts may be talking about two different types of codes used for two different types of research. There are raw data adjustment and infill codes, and then there are model codes. In either case, the results of these researchers are being used to create public law. Therefore the public has a complete right to see this work displayed in all its details, regardless of whether or not we understand it. Those types of restrictions were once used to deny the right to vote.
For temperature purposes, the code has to do with developing a running average and trend line from swiss cheese data infills and adjustments, in other words, the creation of data instead of staying with the typical 999, and the adjustments to data related to the researcher’s idea of contamination. It is the created/adjusted data code (adjustments to raw data and data infill) that is of interest and is apparently not available to the degree that it should be available. The very premise of the research sits wholly on this code. The code is the experiment being done on raw data and should be made available.
Model codes are another thing. While they also should be readily available and are also “the experiment”, these codes try to replicate forcings and feedbacks.
Both types of code should be available for other researchers and the public, to examine and critique, duplicate, or improve.
A point about accusations of arrogance. I bought the damned stuff, the lab, the equipment, the lab assistants, the cost of publication, and I could be asked to pay even more for it if it is used for the creation of laws. So excuse me all to hell but I want to know just what it is I am buying. If that makes me arrogant, fine. I’ll be arrogant, especially when someone else’s hand is in my back pocket. If there ever was a case between who should be and has the right to be arrogant, it would be the paying public, not the researchers. As my grandma used to say to me, you need an attitude adjustment. And shortly thereafter, I got a knuckle on my head. If researchers don’t like that, then get the hell out of the kitchen.
Some commenters (toby, David) seem to suggest that the way how original data are “processed, enhanced and presented” is the IP of the inventor/programmer and not for the eyes of the public or critics. Well, this explains a lot.
Did I miss the “sarc/” somewhere?
Those arguing to distraction that computer code need not be released, commented, easy to read, read by non-programmers…. to support academic papers are missing the essential point. What is required to reproduce and/or replicate the findings of a study is the methodology used to analyse the data and draw the conclusion.
In the absence of fully documented methodology, the code performing the analysis/homogenising/etc should be released. If you do not wish to release your code, you must instead be prepared to explain your methodology fully and in detail in another format on request.
What is NOT acceptable, and is in this circumstance the net result of those arguing that code may be protected by IP, is that NEITHER the code NOR the methodology is provided for effective replication. It is against this background of withholding the entire methodology, or critical portions of methods of analysis, that the call to release code has been made. It is because the claim has been made, that the mechanism is too complex to explain, that those who wish to test replication have requested the programming code instead.
If you will release an academic paper making assertions and drawing conclusions, you diminish to NOTHING the scientific value of your paper at the first sign of resistance to requests to reveal your methodology – if you have not already fully defined it within the pages of your paper which, arguably, you should have done anyway.
Those arguing that code should be for code monkeys and not for scientists to use in the study and subsequent replication of findings ought to be distinguishing themselves from those who would argue that methodology is itself subject to intellectual property rights. The latter argument has no place in academic or scientific research.
Stephen Schneider was a great man and a great scientist. Not perfect, but great.
There has never been a great reason for not releasing the data and the code. It could be (and has been) argued that its ok to withhold if you have future papers in the works. However, when the research and papers are public funded – it all belongs to the public and should be released ASAP.
Actually,iirc, I believe that DR Mann now releases all his data/code for his recent papers – as do many other researchers.
Schneider reminds me of the Wizard in “The Wizard of OZ”,……… except the Wizard helped Dorothy get back to Kansas after he was discovered to be the ordinary man behind the illusion.
Who’s gonna help us get back to “Kansas” now that Schneider has passed.
@Nuke
“Computer code is deterministic. Run the same code with the same data and the results are always the same.”
If identical results from identical input is a design goal then I’d agree with that in principle but there are a great many possible pitfalls in reality. This is especially true when floating point calculations are involved and the underlying hardware, firmware, and ancillary software is not identical.
@chris Long
Bingo!
To distill this story to its critical component, we are left with this unsettling fact: The term “peer reviewed” is meaningless. In fact it’s insulting to make the claim that one has written peer reviewed papers as a means to validate their effort, and utterly vain to state one has peer reviewed other papers. It is a failed process in desperate need of overhaul. I’ve added the term to my list of weasel words.
Stephen Schneider was no doubt brilliant and charming. He was also clearly a post-normal scientist and an apocalyptic grandstander. His jump to Global Cooling in the 1970’s, and then his turnabout jump to Global Warming shows either a lack of scientific discipline in his thinking or fraud in his public presentation of his beliefs, or some of both.
Nuke,
Given the end result wished, I can write in f77 (still have all my notes even) as many nested procedures as is required to accomplish such from any set of numbers and so can ANY other schooled in the late 80’s IT dweeb. The bad bit here appears to be any academical coding requires little if any outside debugging and certainly scarce documentation while industrial coding (my job) is checked here, back in Germany and once again here in good old Cannuckville before it makes its way into any systems or machinery. Bad code in information systems costs coin, in machinery lives and neither is allowable in business. Why should the script kiddies in Academia, whose code is setting the stage for %GDP level spending be afforded any less vigilance?
sphaerica says:
July 21, 2010 at 6:07 am
I thought science was supposed to be about building upon the shoulders of giants. The way you describe it’s more like science is about reinventing the wheel.
An odd fundamental contradiction exists in the CAGW line of reasoning as follows:
Most CAGW proponents say we must act quickly now to save ourselves from ” . . . . the end of the world as we know . . . .” and yet most CAGW advocates actively defend some mainstream climate researcher’s attempts to block/delay access to data, methodology and code from research performed with public funds. The contradiction for these CAGW proponents is that, if it is so important to save the world then it is criminal of them to support researchers who block/delay access to the info.
My thanks to R.E.M. for their great song . . . it is so quotable in the current CAGW area. : )
What would be their purpose in maintaining such an obvious contradiction? This is not a rhetorical question. Seriously.
John
Sphaerica says (among numerous other ridiculous assertions): “The point in that situation is to get to the answer with the least possible effort, not to produce “’great code.’”
I’ve seen code produced with the “least possible effort.” It is shoddy, impenetrable, and incompetent. It is, ipso facto, laden with errors both large and small. We’re dealing with a situation in which trillions of dollars and billions of lives will be risked. Accepting anything less than “great code” is indefensible.
sphaerica says:
July 21, 2010 at 6:07 am
People are entitled to their own intellectual property, and they’re entitled to try to be “the one” to publish that next great ground breaking study. And if some of the information from their last paper is serving as the foundation for the next, then no, they do not and should not have to share it.
=============================
I assume then that you will complain on my behalf, loudly, to Mr. Menne and to Director Karl of NCDC who “borrowed” my data from the surfacestations project when it was 43% complete, over my written objections, and over the objections of my co-authors, ignoring us all and revoking all professional courtesy in order to preempt the paper we are now finishing with the data at 87%?
Seems to me that “climate science” gets a free pass when it suits them.
I look forward to seeing your signed complaint letter to them.
Toby ???
excuse me but WHAT TF are you talking about …
If you publish results and raw data without your code then what is the reviewer actually doing ? making up their own CODE ??? are you kidding me … the work being reviewed is the code you dolt …
More likely it was an attempt to duck a handbag-bashing. (Due to use of the M-word.)