An excerpt from Steve’s post at Climate Audit
Schneider replied that he had been editor of Climatic Change for 28 years and, during that time, nobody had ever requested supporting data, let alone source code, and he therefore required a policy from his editorial board approving his requesting such information from an author. He observed that he would not be able to get reviewers if they were required to examine supporting data and source code. I replied that I was not suggesting that he make that a condition of all reviews, but that I wished to examine such supporting information as part of my review, was willing to do so in my specific case (and wanted to do so under the circumstances) and asked him to seek approval from his editorial board if that was required.
This episode became an important component of Climategate emails in the first half of 2004. As it turned out (though it was not a point that I thought about at the time), both Phil Jones and Ben Santer were on the editorial board of Climatic Change. Some members of the editorial board (e.g. Pfister) thought that it would be a good idea to require Mann to provide supporting code as well as data. But both Jones and Santer lobbied hard and prevailed on code, but not data. They defeated any requirement that Mann supply source code, but Schneider did adopt a policy requiring authors to supply supporting data.
I therefore re-iterated my request as a reviewer for supporting data – including the residuals that Climategate letters show that Mann had supplied to CRU (described as his “dirty laundry”). The requested supporting data was not supplied by Mann and his coauthors and I accordingly submitted a review to Climatic Change, observing that Mann et al had flouted the new policy on providing supporting data. The submission was not published. I observed on another occasion that Jones and Mann (2004) contained a statement slagging us, based on a check-kiting citation to this rejected article.
During this exchange, I attempted to write thoughtfully to Schneider about processes of due diligence, drawing on my own experience and on Ross’ experience in econometrics. The correspondence was fairly lengthy; Schneider’s responses were chatty and cordial and he seemed fairly engaged, though the Climategate emails of the period perhaps cast a slightly different light on events.
Following the establishment of a data policy at Climatic Change, I requested data from Gordon Jacoby – which led to the “few good men” explanation of non-archiving (see CA in early 2005) and from Lonnie Thompson (leading to the first archiving of any information from Dunde, Guliya and Dasuopu, if only summary 10-year data inconsistent with other versions.) Here Schneider accomplished something that almost no one else has been able to do – get data from Lonnie Thompson, something that, in itself, shows Schneider’s stature in the field.
It was very disappointing to read Schneider’s description of these fairly genial exchanges in his book last year. Schneider stated:
The National Science Foundation has asserted that scientists are not required to present their personal computer codes to peer reviewers and critics, recognizing how much that would inhibit scientific practice.
A serial abuser of legalistic attacks was Stephen McIntyre a statistician who had worked in Canada for a mining company. I had had a similar experience with McIntyre when he demanded that Michael Mann and colleagues publish all their computer codes for peer-reviewed papers previously published in Climatic Change. The journal’s editorial board supported the view that the replication efforts do not extend to personal computer codes with all their undocumented subroutines. It’s an intellectual property issue as well as a major drain on scientists’ productivity, an opinion with which the National Science Foundation concurred, as mentioned.
This was untrue in important particulars and a very unfair account of our 2004 exchange. At the time, Schneider did not express any hint that the exchange was unreasonable. Indeed, the exchange had the positive outcome of Climatic Change adopting data archiving policies for the first time.
…
As I noted above, at his best, Schneider was engaging and cheerful – qualities that I prefer to remember him by. I was unaware of his personal battles or that he ironically described himself as “The Patient from Hell” – a title that seems an honorable one.
Read more at Climate Audit
One commentor alledges that he does not like sharing code “because of the time wasted to try to explain it to someone.”
As a part time professional “coder”, your rational for not releasing source code, to put it mildly, stinks of sloth and ineptitude. If I did not document, did not clearly comment, did not make MAINTAINABLE any of my professionally produced “coding”, I would be either fired, not hired on contract, or not able to obtain future contract work once my poor documentation and “idosyncratic” code was revealed. Frankly my Dear, I DO give a damn, and regard ANY work done using computer analysis which does not have “tracibility” as WORTHLESS. (See “Photoshop Job” for an insight into this.)
I’d recommend this fellow try to do some:
1. FDA Approved coding.
2. FAA Approved coding.
3. NRC Approved coding.
I’ve worked with or on all three. A CENTURY from now someone should be able to run my code done with these rules and to make it work, do verifications, and do modifications based mostly on the “code listing” itself.
Max
sphaerica says:
July 21, 2010 at 10:58 am
———————-
sphaerica,
Re your most recent comment, in summary you are saying what science “is” is only worth discussing if it agrees with your concept?
John
Well!
Are you one of those people who makes assertions and then cut off any attempt to discuss and digest? Do you think anybody who disagrees with you is full of hatred? What exactly did I say to you that falls into the realm of “ hatred, innuendo and evil mad scientist conspiracies.” And for the record, I haven’t “spewed” anything since I was a small child.
What really makes this post so unbelievable is you seem to be saying computer code represents the real world, as if a model and the actual climate are indistinguishable.
In the real world, you have to show your work. You can’t just present a paper and say “these are the results” without showing how you got to those results.
You keep forgetting computer models do not output facts.
Get over yourself.
In response to Mike Haseler’s comments about simple terms being complicated by science, I remember the quote of Dr. Howard Hayden who, remarking about how science instead of using ‘man made’ uses ‘anthropogenic’ global warming, said “why use two syllables, when five will do!”
sphaerica says: July 21, 2010 at 10:58 am
sphaerica, you are obviously the one who has no clue how science works. The first step is replication: can I get the same results doing exactly what the scientist claims he did. Following the recipe didn’t work so well for cold fusion. Trying to replicate someone else’s results using your own data and methods and codes is the second step…. but if the results don’t match, you get the sort of response we;ve seen all too often from climate science: “Obviously they were too inept to do it right.”
Hint: Failure to replicate with identical methods sometimes indicates “fraud”. Unwillingness to enable such replication and the kind of disparagement that we have seen coming from you and others strongly suggests it. Science is being corrupted by people just like you.
I would, on principal, but in fact I find most of what you do to be reprehensible and so no, I would not defend you in any way. In principle you may be right (I haven’t looked into the particulars, and don’t particularly care, since I find your work on the surface record to be a valueless distraction), but being right in one small area while being so very wrong in so many others is not suddenly going to get me on or by your side.
But the question at hand stands, and you dodged it: do you believe every bit of knowledge in science should be public property, or do you believe that you were wronged, and that demanding complete 100% transparency is not always appropriate. Do you believe that the way to conduct science is not to nitpick other people’s work, and to demand that they do the heavy lifting for each and every self declared auditor, but rather for scientists to work independently, using a variety of methods and approaches, in order to achieve more robust and defensible conclusions?
REPLY: I asked first, and you dodged with your own question. I believe the way to conduct publicly funded science with full disclosure and open code accountability. Private science should also do so, when their papers are published, as I will do with my surfacestations paper when published. (assuming people like the CRU won’t lobby to keep it unpublished, as we’ve seen in the Climategate emails)
Having big government horn in on a private science project before completion just to CYA is reprehensible, but you are blind to it.
You find me reprehensible, I find that you not only do you hide behind the cowardly comfort of anonymity, as do many of your ilk, you are hypocritical on your own stated positions as well. You have no honor and exemplify the worst of the worst anom-trolls. I didn’t ask you to take sides, only to act on your own position and you showed your true colors. I have no further tolerance for you.
– Anthony
engineering = applied sciences
developmental engineering = engineering research = doing science while trying to achieve a practical end
looks like science to me
[if it looks like a duck, and smell/tastes/sounds/feels/reproduces like a duck, voila . . . a duck]
John
Vukcevic
As always you post an interesting graph. The correlation of the GMF to the colf climate from 1660 to 1700 is particularly intriguing. Is the recent trend of the GMF giving us any hints as to where our climate may be headed over the next few decades?
tonyb
sphaerica says:
July 21, 2010 at 10:58 am (Edit)
And for Pamela Gray, Nuke, Tallbloke, and other angry posters… sorry, I’m not biting. Spew your hatred, innuendo and evil mad scientist conspiracies all you want. I’m not interested, because it has no place in the real world. It’s not even worth discussing.
Lol.
Thanks for playing.
Now naff off.
Re Pamela Gray’s comment at: Pamela Gray says: July 21, 2010 at 8:09 am
I add a hearty “Amen!”
Anthony,
I am a real fan, but especially so when you bring up the anonymity topic.
Thank you and your team for all the work it takes to make this place an independent venue. You do honor to the long standing (ancient) traditions of open forums.
John
To Mr. Toby,
With all respect to your position, a Sophmore in their first numerical methods class could explain exactly why you need to present your code as part of a review. If everything is perfect, but a count is off (ie, the infamous “count++ should have been ++count”), a mathematical function can give reasonable-looking-but-wrong answers. Small deficiencies within code that takes weeks to write and debug are often key to problems in a calculation. It is absurd to think that a reviewer should be able to recreate the calculation, run it, debug it, and then decypher that the problem was with the original and not the recreation in their side-position of reviewing papers. If a statistician is to properly review the work, they need to review the actual code, not just read a summary of the method and then check a “Looks OK” box. I cannot tell you the number of times I have seen people describe the code, then look at it and find that due to bugs it behaves nothing like the summary. Now, if your code is idiosyncratic and uncommented, then that is no different than writing your laboratory results in gibberish shorthand. If you would not fail a student for incomprehensible notes in the lab, then I must think less of you. If you expect a different standard from yourself, you have no place performing research.
Sapherica, Logic 101: If you cannot support a statement then you should not make a statement. While your arguments about confidentiality hold in a corporate environment, it does not apply to peer-reviewers, who can and should be subject to confidentiality and proper-use agreements.
tonyb says: July 21, 2010 at 11:53 am
Is the recent trend of the GMF giving us any hints as to where our climate may be headed over the next few decades?
Professional climate experts with huge funds and the ‘Cray’ supercomputers do predicting, I do it for fun. I wouldn’t put any bet on either.
“With high hope for the future, no prediction is ventured.” A.L.
Times have changed since.
Sphaerica shows little or no experience or knowledge of science, unfortunately.
Indeed, and there are many examples of exactly that happening in science. If Sphaerica actually knew his/her scientific history, she/he would be aware of it.
One of the most famous examples is Prosper-René Blondlot’s discovery of “N-rays”. If you have never heard of them before, it is because they do not exist: but their very existance is an object lesson in scientific experimentation.
Blondlot conducted an experiment in which he claimed to discover a new form of radiation which he called N-rays. Of course, being a new discovery in science, many people tried to replicate the experiment. But the replications were ambiguous: many labs claimed to replicate the results (some even claiming the discovery for their own), others failed to replicate them. Naturally, disputes arose as to whether those that failed to replicate the results had correctly executed the experiment – since so many others had claimed success.
The issue caused much fuss at the time. It was resolved when one of those sceptical of the results – US scientist Robert Wood went to Blondlot’s lab to monitor his experiment. In “auditing” Blondlot’s experiment, he found that by double-blinding the experiment in Blondlot’s lab, using Blondlot’s experimental setup and staff, the positive results disappeared. By showing unequivocally that Blondlot’s experiment failed – rather than his own set up, there was no ambiguity, no room for doubt. The issue was resolved in a way that replication could not have achieved.
Note that this is a great example of science operating in exactly the way Sphaerica said it doesn’t. Of course, Blondlot’s experiment was a fairly simple setup. There are far more scope for errors in even a simple experiment due to the complexities in software code, which can involve a sequence of statistical analyses, which are often not fully or correctly documented.
Perhaps Sphaerica should consider a little more humility in his/her claims until learning a bit more about the history of scientific research.
Point to Ponder
There is no such thing as “private science”. Lord alone knows what those guys in private, commercially-funded labs are doing, but if their processes and methods are not open to public scrutiny, then it is not science. The public nature of science is in many ways its most critical aspect. Kind of reminds me of those smirking Extenz commercials on late night TV, “this is real science…” yeah.
John Whitman says:
July 21, 2010 at 10:26 am
“Not being computer programmer oriented, I have a question.
Aren’t there computer software industry standards like similar to mechanical engineering’s ASME, AWS, ANSI? If so, do any standards govern the areas of computer software development/control?”
Yes, in safety-related applications like railways, avionics, power plants, esp. nuclear. In the EU, you have SIL0 to SIL4 , safety integrity levels 0 to 4, and the higher levels require you to choose a coding style that you are able to justify – the easiest way to do this is to pick an industry standard set of guidelines like MISRA, or in Germany for railways, something like MÜ 8004.
You can define your own coding style but you will have to document it and you will have to convince the authorities that it’s state of the art what you’re doing.
Pamela Gray says:
July 21, 2010 at 8:09 am
The posts may be talking about two different types of codes used for two different types of research.
…. I bought the damned stuff, the lab, the equipment, the lab assistants, the cost of publication, and I could be asked to pay even more for it if it is used for the creation of laws. So excuse me all to hell but I want to know just what it is I am buying. If that makes me arrogant, fine. I’ll be arrogant, especially when someone else’s hand is in my back pocket. If there ever was a case between who should be and has the right to be arrogant, it would be the paying public, not the researchers. As my grandma used to say to me, you need an attitude adjustment. And shortly thereafter, I got a knuckle on my head. If researchers don’t like that, then get the hell out of the kitchen.
_________________________________________________________________________
I agree. If it can not be replicated it is not science.
If it is paid for with my tax dollars then I own it</b. not the scientist who did the research and therefore I have the right to see it… ALL of it.
I would like to add to that. If a scientist is going to be that protective over his research he should not be publishing in the first place. Get out of academia and go work for industry.
What is the definition of publication any way?
Definition
Publication is the act of offering something for the general public to inspect or scrutinize. It means to convey knowledge or give notice. Making something known to the community at large, exhibiting, displaying, disclosing, or revealing….
Notice that scientists? Publication is the act of offering something for the general public to inspect or scrutinize. Therefore “Climate Scientists” are not even meeting the definition of publishing much less the definition of science they only meet the definition of propaganda.
Definition
propaganda: In general, a message designed to persuade its intended audience to think and behave in a certain manner. Thus advertising is commercial propaganda. In specific, institutionalized and systematic spreading of information and/or disinformation, usually to promote a narrow political or religious viewpoint. Originally, propaganda meant an arm of the Roman Catholic church responsible for ‘de propaganda fidei,’ propagation of the faith. It acquired negative connotations in the 20th century when totalitarian regimes (principally the Nazi Germany) used every means to distort facts and spread total falsehoods.
Now compare that to Schneider statement:
“We need to get some broadbased support, to capture the public’s imagination. That, of course, entails getting loads of media coverage. So we have to offer up scary scenarios, make simplified, dramatic statements, and make little mention of any doubts we might have. This ‘double ethical bind’ we frequently find ourselves in cannot be solved by any formula. Each of us has to decide what the right balance is between being effective and being honest. I hope that means being both.”
Almost everything in this science of CAGW is too overstated – in 1979 Stephen Schneider was telling us the CO2 will rise +20% at the end of century and in middle of 21st century “will double” – http://www.youtube.com/watch?v=pB2ugPM0cRM .
In fact now we have 2010 and CO2 have risen 13.7% since 1979, 5.5% since 2000. So he overstated the figure 2.4 times than actually was the reality.
It reminds me the fishermen who show using their hands how big the fish they caught was to win attention. It is more like an activism than a serious science…
Robert,
Sure, I can see that there must be independent checks and balances in a scientific process for it to remain objectively focused. That can be done privately. I see no reason why individual people involved (I purposely didn’t say scientists) in a well set up process could not all be private citizens funded privately and even confidentially. Why public?
Are you implying by “public” that governments and newspapers are necessary for the scientific process?
NOTE: In the case of using government funding for areas like climate science I see that the process needs to be public.
John
Ecotretas says:
July 21, 2010 at 2:02 am
It is enlightening to watch the video of a Channel 4 TV programme called ‘The Global Warming Conspiracy’ on Ecotretas’ website.
It was made in 1990 and it is quite obvious that climate science has not progressed at all in the intervening 20 years. The names have changed: in the film Wigley was running the CRU, Houghton was at the UK Met office. Michaels, Lindzen, Spencer, Schneider and a few others who have dropped from the scene (probably because of their sceptic views) definitely look 20 years younger but still say exactly the same things as now.
In fact, some things have got worse. When asked difficult questions the tone of the reply was somewhat less angry 20 years ago.
It is depressing how little has really changed – despite billions of dollars in funding there is no more certainty in the hypothesis of global warming than there was 20 years ago.
Robert E. Phelan says:
July 21, 2010 at 1:12 pm
——————
Robert,
Sure, I can see that there must be independent checks and balances in a scientific process for it to remain objectively focused. That can be done privately. I see no reason why individual people involved (I purposely didn’t say scientists) in a well set up process could not all be private citizens funded privately and even confidentially. Why public?
Are you implying by “public” that governments and newspapers are necessary for the scientific process?
NOTE: In the case of using government funding for areas like climate science I see that the process needs to be public.
John
[snip – calling people sociopaths won’t help]
toby says:
July 21, 2010 at 12:25 am
“What is in question is replication, and a critic should be able to write his or her own code in order to test results with the same data.
The code you write yourself when doing a paper is not for commercial use and not user-friendly. It is usually idiosyncratic and uncommented (or at least poorly commented). Giving it to somebody implies spending valuable time explaining all the wrinkles as well – a waste of time with someone who should be able to do the job themselves.”
Crap code is crap code.
Some basic fundamentals of good programming.
You should always write your code to the best standards, because even throwaway stuff will get reused if it does anything remotely useful.
It is not quicker to write sloppy code, because you never get it 100% right the first time, and debugging always takes longer.
You won’t have time to go back and write it properly, so do it right right from the start.
If your code is not good enough for public display then it probably does not work properly anyway.
Yes, if you publish a complete formula then someone else can write their own program to replicate the results, but you have just doubled the possibility of bugs affecting the results.
You don’t see the Presenters at Top Gear creating vehicles from manufacturers blueprints so that they can replicate the performance of a new vehicle. The manufacturer provides a complete and working example to be test driven.
Its about time that the academic community were held to the same standards that the rest of the world work to.
Are we discussing the work or the person?
My condolences to his family.
sphaerica says:
July 21, 2010 at 10:58 am
“Smokey, Chris Long, Scott B., vigilantfish, and all the others…
You still don’t understand how science works.”
———–
I hope I understand how science works – I have a Ph.D. and a professorial career in exactly that field: the study of how science works ( i.e. history and philosophy of science. ) Try again.