In my opinion, this is a testament to Steve McIntyre’s tenacity.
Via the GWPF: At Last, The Right Lesson From Climategate Fiasco
A diverse group of academic research scientists from across the U.S. have written a policy paper which has been published in the journal Science, suggesting that the time has come for all science journals to begin requiring computer source code be made available as a condition of publication. Currently, they say, only three of the top twenty journals do so.
The group argues that because computer programs are now an integral part of research in almost every scientific field, it has become critical that researchers provide the source code for custom written applications in order for work to be peer reviewed or duplicated by other researchers attempting to verify results.
Not providing source code, they say, is now akin to withholding parts of the procedural process, which results in a “black box” approach to science, which is of course, not tolerated in virtually every other area of research in which results are published. It’s difficult to imagine any other realm of scientific research getting such a pass and the fact that code is not published in an open source forum detracts from the credibility of any study upon which it is based. Articles based on computer simulations, for example, such as many of those written about astrophysics or environmental predictions, tend to become meaningless when they are offered without also offering the source code of the simulations on which they are based.
The team acknowledges that many researchers are clearly reticent to reveal code that they feel is amateurish due to computer programming not being their profession and that some code may have commercial value, but suggest that such reasons should no longer be considered sufficient for withholding such code. They suggest that forcing researchers to reveal their code would likely result in cleaner more portable code and that open-source licensing could be made available for proprietary code.
They also point out that many researchers use public funds to conduct their research and suggest that entities that provide such funds should require that source code created as part of any research effort be made public, as is the case with other resource materials.
The group also points out that the use of computer code, both off the shelf and custom written will likely become ever more present in research endeavors, and thus as time passes, it becomes ever more crucial that such code is made available when results are published, otherwise, the very nature of peer review and reproducibility will cease to have meaning in the scientific context.
More information: Shining Light into Black Boxes, Science 13 April 2012: Vol. 336 no. 6078 pp. 159-160 DOI: 10.1126/science.1218263
Abstract
The publication and open exchange of knowledge and material form the backbone of scientific progress and reproducibility and are obligatory for publicly funded research. Despite increasing reliance on computing in every domain of scientific endeavor, the computer source code critical to understanding and evaluating computer programs is commonly withheld, effectively rendering these programs “black boxes” in the research work flow. Exempting from basic publication and disclosure standards such a ubiquitous category of research tool carries substantial negative consequences. Eliminating this disparity will require concerted policy action by funding agencies and journal publishers, as well as changes in the way research institutions receiving public funds manage their intellectual property (IP).
=========================================
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
joel shore says:
“I have given links to Mann’s 2008 code that is so completely public that even McIntyre can’t find a nit to pick in that regard.”
If you don’t mind, I will take Steve McIntyre’s word for it.
Not yours.
Smokey :”I don’t understand that at all. Could you please re-phrase, with context? Thanks.”
Read the previous comments re Popper. The context basically is that I think that Willis is too hung up on falsification.
jimmi,
Falsification is absolutely essential to the scientific method. Anything less is post-normal science; AKA: pseudo-science.
Except ,Smokey, were we to apply Popper’s own criteria to Popper’s theory, it would be falsified, though that that would be a paradox….
Science, I regret to inform you, proceeds not by Popper’s ideas in practice, but by the good old incremental improvement model, coupled with the occasional Kuhn-style paradigm shift. And the relevance that observation is that I hold that the way to deal with examples like Mann is not to spend a decade banging on about falsification, but instead just to publish a better version.
jimmi,
You do realize that Kuhn was a post-normal guy, don’t you? Sorry, but I don’t think post-normal science is science at all. Testability is essential. If something [like AGW] is not testable, it is not science, it is belief; a conjecture. So I guess we’ll just have to disagree about this. Even though it looks like we’ve both got Michael Mann’s number.
jimmi_the_dalek says:
April 21, 2012 at 8:08 pm
Dang, you’re in a hurry. I have no idea if I agree with you, since neither of us have defined what we mean by “other ways to make progress”. From what you say below, we may not agree at all.
Indeed … and the reason ideas get replaced is because they are falsified. Not necessarily 100% falsified root and branch, Newton’s ideas weren’t thrown out by Einstein’s insights. But the applicability of Newton’s laws in all situations was certainly falsified. And that’s why it Newton’s ideas were replaced in those situations by Einstein’s work.
Which is why I said above that we may not agree. What you have done is just give another example of falsification, not another way to make progress.
I would put out, as another way to make scientific process, the discovery of graphene. There was nothing that needed to be falsified to make that discovery, so totally new discoveries are another way to make scientific progress.
Please note, however, that the new discovery must be falsifiable to endure. If you say “I’ve discovered a new arrangement of carbon, but I’m not going to tell you what it looks like or what its properties are”, that is not falsifiable. On the other hand, “I’ve discovered a flat, single-atom thick hexagonally structured form of carbon” is very falsifiable … and has not been falsified.
I don’t understand what you are getting at here. Things are never any more than provisionally “true”, they are only valid until they are falsified. Which may be never, or may be tomorrow. But this doesn’t mean that “all of science … is incorrect”. For example, it’s not “incorrect” that 2 + 2 = 4, or that hydrogen is the simplest element, or that nuclear fission can be initiated by certain specified procedures. So I don’t understand what you mean that “all of science … is incorrect”. That seems like far too broad a statement to be true.
For your final comment, I have to back up to provide context, as you have not quoted your claim that started that part of the discussion. You had said:
The conversation continued with my response and your reply:
If I understand you correctly, you are asking if using incorrect equations to make a prediction is whatever you are calling “doing science” … I haven’t a clue, because I don’t know what you are calling “doing science”.
I’d say in general that using known incorrect equations is just plain dumb, but I have no idea whether it fits into whatever you are calling “doing science”.
In any case, why not give up the Socratic method of asking impenetrable questions, and just say what’s on your mind?
Because I still haven’t a clue what you are getting at with your questions. What does using equations which are known to be incorrect have to do with science? Who would do such a foolish thing as knowingly using incorrect equations? I don’t get it.
w.
jimmi_the_dalek says:
April 21, 2012 at 9:34 pm
The problem with that theory is that the “good old incremental improvement model” proceeds because whatever was “incrementally improved” was falsified. See my discussion of Einstein and Newton directly above.
Newton’s Laws were “incrementally improved” by Einstein’s ideas because Newton’s Laws were falsified in certain unusual situations. This was tested using the transits of Mercury, for which Newton’s Laws didn’t work … in other words, in those certain situations Newton’s Laws were falsified.
If that had not occurred, if Newton’s Laws had worked perfectly to predict Mercury’s transits, Newton would not have been falsified, and Einstein’s ideas would have been thrown in the trash.
In other words, your “good old incremental improvement model” depends completely on falsification.
w.
joeldshore says:
April 21, 2012 at 8:34 pm
Thanks, Joel. Since S&C were not forced or ordered to hand over their code, it was voluntary, so no spin there. I’ve heard nothing from RSS saying they asked for other parts of the code and didn’t get it. Nor have I seen anything from Christy about how much code they gave to RSS, and you provide no cites. So I’m not sure what you call spin, what I said seems to be all true to me. I have not made any overarching claims about any of that.
But this is because I have little interest in Spencer and Christy’s code. I can make sense of Mann’s code and understand it. But what S&C are doing to transform satellite observations into temperatures is another universe from where I work, so perforce I must leave it to folks like the guys at RSS. Obviously, since they found an error in S&C’s code, they must understand it, so I leave it in their hands.
You seem to think I should take sides in every dispute, even parts of the grand scientific adventure in which I have no interest … sorry, not my job. The world is full of issues, every man has to choose which battles to fight, and this is not one I care to fight, I lack both information and interest.
If this issue is as important to you as you claim, if you want S&C to do something specific with their code, how about you ask them? Or alternatively, since you haven’t asked them, why are you acting like this issue is so important to you?
I have no idea why you think it is my business to take a position on S&C and their code, when I know very little about it other than that they gave enough of it voluntarily to RSS for the RSS guys to find the error that they suspected. You seem to want to make that into a crime … but what does that have to do with me?
Seriously, Joel, you are obsessing about my position on S&C’s code, and I don’t have one, I don’t have the information to form one, and I don’t particularly care about the issue. I have far too many battles to fight without taking on the ones you think are important. If there were a blowup where RSS was demanding the code and S&C were refusing them, I might take a side if the issues were clear.
But that’s not the case. S&C voluntarily gave enough code to RSS to allow them to find the error they suspected, RSS isn’t making any noise about being denied code, and I have neither the information nor the interest to take some stand on the issue just to please you. So sue me …
w.
PS—Above, you claim equivalency between Mann and S&C, saying that both had been falsified. The difference is, when S&C were shown that their code contained an error, they fixed it. When Mann was shown that his code contained an error, he denied it. Hardly equivalent … heck, Mann is still using Tiljander upside-down …
Willis Eschenbach says:
Mann wasn’t “forced” to hand over his code either. And, he certainly wasn’t forced to release all of the code for his 2008 paper.
Here is what Christy says ( http://climateaudit.org/2006/10/02/christy-on-source-code/ ):
So, that is Christy, putting his best face on what they did, admitting that they have not allowed RSS access to the complete codes and using the excuse that they haven’t seemed to need it in order to basically figure out what Spencer and Christy have done from their description of their methods. Sound familiar?
And, I am not saying what Spencer and Christy did was wrong or that we should be calling them frauds. But, then I am not the one who is making broad pronouncements about how publishing papers without the full release of code is not science, etc., etc.
Fine…So, you personally don’t have such a strong interest in it. However, given that you have used full release of code to make broad pronouncements about scientists and to say whether something is “science” or “anecodote” (even in fields that you know absolutely nothing about, I might add), it is strange how you are so quiet on the issue of the full release of Spencer and Christy’s code because it is not quite in the same subfield of the general field of climate studies that you are most interested in.
That doesn’t seem to stop you in other situations.
That is because RSS doesn’t think that they are entitled to every shred of code that they may want to look at, because they are engaged in real science rather than grandstanding.
Mann has moved on. The best objective scientific assessment of Mann’s codes and temperature reconstructions in general was provided by the NAS panel convened on the matter. And, they did not say that Mann’s results were erroneous, although they did point out some pitfalls of the method that Mann originally used (and no longer uses) and identified some other issues that need to be addressed to get greater certainty (and better quantification of uncertainty) in the reconstructions, particular before 1600. Mann’s 2008 paper is an attempt to deal with these issues.
I am surprised that you are so tempted to use this sort of talking point. The method that Mann used decides the orientation of the proxy automatically on the basis of the correlation with the historical temperature record over a certain time period. Yes, the Tiljander proxy ended up upside down from the way in which the originators of the data apparently thought the data would correlate with temperature. So, as a result of this (and the belief of those authors that other anthropogenic interference might have contaminated the proxy over the period it seemed correlated with the temperature record), Mann also repeated the analysis with this proxy removed.
@Willis Eschenbach (April 21, 2012 at 1:55 pm)
Willis, there’s important work to do.
Why do you waste your time chasing red herrings and attacking straw men?
The only possible reason I can see:
Your interest in politics FAR exceeds your desire to explore & know nature.
Rather than succumb to incessant tabloid-style fascination with Dr. Michael Mann, the “hockey stick”, “climategate”, etc., you could redirect your focus productively by helping out with efforts to understand natural variability. See for example:
http://wattsupwiththat.com/2012/03/30/open-thread-weekend-9/#comment-940636
The incessant whining for code spoonfeeding makes the community look quantitatively weak. If the political & quantitative education systems have failed an individual, that individual has the option of attempting to take personal responsibility:
http://en.wikipedia.org/wiki/Peter_Principle
We may one day lose our freedom for the simple reason that we wasted it.
Paul Vaughan says:
April 22, 2012 at 8:09 am (Edit)
So according to you, you think Science magazine should not be interested in supporting transparent science because it’s “politics” …
My friend, your post is truly way, way out in left field. I have no idea why you think supporting transparent science is “politics”, but it’s not. Transparency is a crucial part of science.
“The community”? Science magazine is “the community” in your curious geography? What planet are you living on?
How does requiring code before publication make anyone “look quantitively weak”? You’ve lost the plot completely.
w.
Joel, you seem obsessed with Spencer and Christy’s code. And that’s fine. Every man chooses what is important to him, every man decides which battles he wants to fight.
What is bizarre is your continued insistence that because you care about their code, that I should care about their code. I don’t. Simple as that. It’s not an issue in my world.
And what is truly over the top is that despite your claimed deep and passionate concern about the matter, in the real world you don’t care enough about S&C’s code to get up off your dead … chair and do one single solitary damn thing about it.
I’m sorry, Joel, but I’m not going to fight your fights. I have plenty of my own. I don’t think there’s a problem with the S&C code. You claim to care about it, but instead of actually doing something about it, you keep insisting that I should carry your load, that I should share your concern, that I should go into battle over this issue.
Sorry, my friend, but you have to conduct your own wars. I have neither the time nor the interest to fight them for you, I have battles of my own to fight. If you truly want to see more of their code, then I suggest that you go ask them for it. Because it’s useless to be talking to me about them not doing what you think they should be doing.
Me, I agree with Steve McIntyre, who commented in your citation,
Now, it seems that Spencer and Christy’s effort is not enough for you, you want them to do more. And that’s a legitimate point of view, although I don’t share it myself. But since you hold that position, OK, fine, then GO DO SOMETHING ABOUT IT.
Because bitching about it to me is meaningless—I don’t care about it, and I already have more battles on my plate than I know what to do with.
If you don’t have the balls to fight your own fights, Joel, don’t come complaining to me and insisting that I should do something you are unwilling to do yourself. That’s your business, not mine. If it truly is an issue to you as you keep claiming, then man up and confront Christy about it.
And if you’re not willing to man up about it, then my suggestion is that you shut up about it, because trying to bust me for not something that you are unwilling to do yourself just makes you look like a hypocrite.
w.
Willis Eschenbach says:
What you don’t get is that it’s not my fight…It is your fight. I am not the one going around making bombastic statements about people being frauds and people engaging in anecdotes instead of science because they aren’t publishing their code along with their papers. I am just the one who is asking YOU not to be a hypocrit and, if you want to make such bombastic statements, then apply them even to people who you might like.
Or, better yet, admit that your statements are out-of-line. That would be a rather smart thing to do, as Paul Vaughan notes, even from a pragmatic point-of-view. Calling respected scientists “frauds” or claiming that papers published without code constitute “anecdotes” is a good way to make sure that the larger scientific community doesn’t take you seriously.
Oh, and by the way, I was thinking a little bit more about your statement regarding the difference between Mann and Spencer & Christy being that the latter have acknowledged their mistakes. I already noted some other issues I have with that claim but I will also note, although Spencer and Christy have eventually acknowledged mistakes with the satellite data record, they have tried to downplay them (e.g., they have said things that seem to imply that the mistakes made minimal difference when in fact it is easy to check that they are responsible for about half the difference in the LT satellite trend they claimed in the late 90s vs the current trend they claim, with the other half being due to the longer data record).
Furthermore, on other issues, they have not come clean:
(1) To my knowledge, Christy has never forthrightly acknowledged the huge statistical blunder they made in the Douglass et al paper ( http://onlinelibrary.wiley.com/doi/10.1002/joc.1651/abstract ) by comparing the data record to the models but using the standard error rather than the standard deviation as a measure of the uncertainty in the model predictions.
(2) To my knowledge, Spencer has never acknowledged the huge mathematical blunder he made in a post here (pointed out by tamino, but pretty obvious to anyone once it is pointed out) in which he tried to argue for evidence that the rising CO2 trend might not be due to humans.
Few scientists are as forthright as they should be about acknowledging errors. I guess it is a general defect of human nature.
joeldshore says:
April 22, 2012 at 2:35 pm
So you now think that you are in charge of deciding what is my fight and what is not? Really? Your ego has expanded to the point that you are now deciding what fights other people should fight? Man, you AGW folks know no bounds.
My friend, you’ve jumped the shark entirely. Every man gets to choose which fights he wants to fight, and which fights are not important enough to be worth the candle … and no man can decide that for another man.
Come back when your ego is no longer large enough to require its own postal code, and we can discuss this … in the meantime, consider which of us is claiming loudly that this is a huge issue. That would be you. I don’t think it’s a big issue at all. If it’s a huge issue for you, that’s fine with me.
But saying that you, the almighty Joel, have decided that it has to be a huge issue for me as well?
Sorry, never gonna happen. You are welcome to fight with S&C over the availability of their code if that is as important as you claim, or not as you see fit. That’s up to you … although it’s odd that you refuse to do anything about it while simultaneously claiming that it is hugely important.
But it’s not a fight I’m interested in. Me, I have other fights to fight, and there’s no way you get to pick which fights those are.
w.
PS—Christy said in 2010:
So Christy is working with NOAA to get it set up so that you and I can not only see the code, but actually run it … but noooo, that’s not good enough for Joel, he wants the code RIGHT NOW or he’ll throw a tantrum and accuse me of mopery on the skyways and other unspecified crimes.
In any case, NOAA is handling it, and the IT/programming team hasn’t gotten it done yet. Go complain to John Bates of NOAA if you don’t like that.
Now, I’m sorry if Christy’s actions in that regard don’t satisfy you. I regret that’s not enough for you, that you want something different.
But you don’t get to decide if that’s enough for me, Joel, and all of your petty bitching about it and all your nasty accusations about me don’t change that one bit.
Oh, yeah, Joel, you make this claim:
joeldshore says:
April 22, 2012 at 2:35 pm
Well, let’s see. They’ve said:
and
and
and
and regarding the error discovered by RSS they say
and a few more errors
and
and
Now Joel, perhaps you can point out in there just exactly where they “have tried to downplay” their errors. Seems to me that it would be hard to be more honest about the exact nature and size of the errors than they have been. So please indicate for us just where you claim they are downplaying the size of the errors.
Note also that they reported these errors clearly and openly, rather than hiding or dissembling about them.
You also falsely claim that they have “eventually acknowledged mistakes”, when in fact they have reported them in detail in a timely manner. You’re just making up nasty accusations and hoping they stick.
In short, I don’t have a clue why you are hating on them regarding their handling of errors. If Mann and others were as open about their errors as S&C have been, life would be much simpler.
w.
John Christy, from here:
In his testimony, Christy has clearly laid out the issues and the decisions and the outcomes regarding his experience with the sharing of code. He is strong and emphatic about the advantages of openness and transparency.
And that, Joel, is why the availability of S&C’s code is not an issue for me … yes, as yet there’s no URL available where I can inspect either their six programs that run sequentially on three separate machines or the terabytes of data. But John Christy is an ethical scientist who understands the importance of the sharing of both. And as a result, it does not concern me.
You keep insisting I have some imagined obligation to speak out against his practices.
Unfortunately, from my point of view, there’s nothing I can either teach him or reproach him with regarding transparency. What would I say?
w.
@Willis Eschenbach (April 22, 2012 at 10:24 am)
I will leave you to your unproductive pursuits.
Paul Vaughan says:
April 23, 2012 at 6:47 am
Thank goodness, I thought you’d never leave.
w.
adiós – ’til we meet again…
http://i40.tinypic.com/16a368w.png
I think another reason scientists hesitate to publish their code is that *they do not want to support it*. Sometimes, people think the original authors of code are required to solve their problems. The “no warranty” clause of the GPL is an important one. The culture of “patches welcome” hasn’t really made it into academic science, where it’s more likely to be seen as rude that you’d edit somebody else’s work.