In my opinion, this is a testament to Steve McIntyre’s tenacity.
Via the GWPF: At Last, The Right Lesson From Climategate Fiasco
A diverse group of academic research scientists from across the U.S. have written a policy paper which has been published in the journal Science, suggesting that the time has come for all science journals to begin requiring computer source code be made available as a condition of publication. Currently, they say, only three of the top twenty journals do so.
The group argues that because computer programs are now an integral part of research in almost every scientific field, it has become critical that researchers provide the source code for custom written applications in order for work to be peer reviewed or duplicated by other researchers attempting to verify results.
Not providing source code, they say, is now akin to withholding parts of the procedural process, which results in a “black box” approach to science, which is of course, not tolerated in virtually every other area of research in which results are published. It’s difficult to imagine any other realm of scientific research getting such a pass and the fact that code is not published in an open source forum detracts from the credibility of any study upon which it is based. Articles based on computer simulations, for example, such as many of those written about astrophysics or environmental predictions, tend to become meaningless when they are offered without also offering the source code of the simulations on which they are based.
The team acknowledges that many researchers are clearly reticent to reveal code that they feel is amateurish due to computer programming not being their profession and that some code may have commercial value, but suggest that such reasons should no longer be considered sufficient for withholding such code. They suggest that forcing researchers to reveal their code would likely result in cleaner more portable code and that open-source licensing could be made available for proprietary code.
They also point out that many researchers use public funds to conduct their research and suggest that entities that provide such funds should require that source code created as part of any research effort be made public, as is the case with other resource materials.
The group also points out that the use of computer code, both off the shelf and custom written will likely become ever more present in research endeavors, and thus as time passes, it becomes ever more crucial that such code is made available when results are published, otherwise, the very nature of peer review and reproducibility will cease to have meaning in the scientific context.
More information: Shining Light into Black Boxes, Science 13 April 2012: Vol. 336 no. 6078 pp. 159-160 DOI: 10.1126/science.1218263
Abstract
The publication and open exchange of knowledge and material form the backbone of scientific progress and reproducibility and are obligatory for publicly funded research. Despite increasing reliance on computing in every domain of scientific endeavor, the computer source code critical to understanding and evaluating computer programs is commonly withheld, effectively rendering these programs “black boxes” in the research work flow. Exempting from basic publication and disclosure standards such a ubiquitous category of research tool carries substantial negative consequences. Eliminating this disparity will require concerted policy action by funding agencies and journal publishers, as well as changes in the way research institutions receiving public funds manage their intellectual property (IP).
=========================================
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
>>
Windchaser says:
April 19, 2012 at 11:53 am
Why not just write your own code?
<<
I’ve read some naive statements on this thread, but this one takes the cake.
Let’s say, for grins, that I wrote a program that thinks–it passes the Turing test with flying colors. You want to see MY code? Hey, why not just write your own?
Besides, you just want my code so you can prove me wrong.
/sarcasm off
Jim
Windchaser says:
April 19, 2012 at 3:19 pm
Fallacy of the Excluded Middle. You left out … or
3) the code contains bugs, or
4) the code contains errors in logic unknown to the authors (see Michael Mann and the Hockeystick), or
5) the authors are flat-out lying about what they did, or
6) the authors described what they thought they did, but in fact they did something else, or
7) Harry of the read-me file has “tuned up” the code without the knowledge of the authors, or
8) the authors are shading the truth for business reasons, or
9) the code has accidentally become corrupted in the process of copying, or
10) the users of the code are not fully conversant with the limitations and oddities of the code, or
11) bad input has led to Garbage In, Garbage Out, or
12) the code is machine-specific, and produces different answers on different machines, or …
You’re back on to “trust me” … but science is not about trust. It is about verification. The only way to verify that the authors have done what they think they have done is to reveal the code.
Let me repeat what the article said, and remind you that the authors are not unknown like yourself. Unlike you, they signed their names to the study, and they are from the following institutions:
Harvard Medical School, Boston
School of Law, University of California, Berkeley
Lawrence Berkeley National Laboratory, Berkeley
Argonne National Laboratory and University of Chicago
University of California, San Francisco, San Francisco
University of Washington and Howard Hughes Medical Institute
Here’s what they said again (emphasis mine):
You see that part about “… carries significant costs, yields few benefits, and is inconsistent with accepted norms …”?
Now you are here, a random anonymous internet popup, to claim there are no costs, that there are lots of benefits, and that it is in fact consistent with scientific norms … sorry, but you’ll have to do better than that.
Science mag sees the issues with code withholding. The six authors see the issues. Science magazine, PLoS, PNAS, and the Journal of Biological Sciences have changed their policies because of the issues. Most of the folks posting here see the issues.
When I was a kid on the cattle ranch, they used to say “If one man calls you a horse, laugh it off. If two men call you a horse, think it over. If three men call you a horse … buy a saddle.” True to your alias, you’re chasing the wind on this question, my friend, you are clinging to a rapidly disappearing past, a time when scientists could be trusted and procedures were simple enough to describe in English. That time is gone forever.
w.
Willis Eschenbach says:
I think that the editors of Science are, like most journals, wrestling with what the require authors supplementary materials authors submit along with their papers and thus feel it is important to have an active and vigorous discussion about where to draw the line. I don’t think that the editors necessarily agree that the line should be drawn where the authors of that particular place say it should be drawn. I would need a clearer endorsement of this from the editors before I believed it represented their views.
That editorial is rather strange to me in that they spend most of it talking about the issue of data and then throw in this one sentence that, if interpreted strictly might seem to imply that essentially all code must be submitted. I would be rather surprised, however, to hear that Science is actually enforcing the strictest interpretation of that vague sentence, as it would raise all sorts of interesting questions about what papers they could publish and what they couldn’t. Basically anything using any sort of proprietary code would be out if one adopted the strictest interpretation of that one vague sentence…and I imagine one could find all sorts of articles actually published in Science since that editorial was written that don’t comply with a very strict interpretation of that sentence. (And, as I recall, is it not true that you or others have complained that Science is not adhering to their own policy because they are in fact not enforcing that requirement in the strictest sense that you would like to see it enforced. Or, am I remembering something involving a different issue or journal?)
Willis Eschenbach says:
barry gave you the link that I was thinking of. You and McIntyre seem to have some vague complaints about why this isn’t good enough…which are great at distracting everybody from the fact that Mann has released more code than any reasonable person could possibly ask for and almost as much as a not-so-reasonable person like McIntyre could ask for (so much, in fact, that even McIntyre has to go back to papers over a decade old to find anything to complain about!) In the meantime, we have not a shred of code provided by Spencer and Christy. And, despite this, Mann is pillaried and Spencer and Christy are considered minor deities in the community! Go figure.
Jim Masterson thinks that Windchaser is being “naive” for suggesting that you should write your own code, but actually that is what a proper scientific test requires.
Think experiments : you want to check the Michelson-Morley experiment – do you say it is impossible because you do not have Michelson’s interferometer? No, you build a better one.
A true check of an “experiment” which depends on a computer program does not require that you run the same calculations with the same program (and on the same machine/hardware/operating system/system libraries etc etc). Frankly that is a very limited check. A true check would insist on using an independent program with the same functionality to see if it gave the same answers.
I am asked to referee papers in a branch of physical chemistry which often using computer generated results. Sometimes I see results which do not make sense. The authors claim they have used program X. Do I check them using program X? No of course not, that would prove nothing. I use program Y instead. If they then agree I conclude that my intuition that the results were wrong was lacking. If they disagree, I suggest to the authors that they may re-consider the methodology. That is the proper way to do a check. Insisting that they produce the code is pointless because who is going to look through a million lines of C++ or whatever. You test programs by comparing the results with another independent program.
joeldshore says:
April 19, 2012 at 5:25 pm
That’s freakin’ classic. For years and years, Mann refuses time after time to provide his code. Finally, after almost a decade, he releases part of it … and then you whine that McIntyre is discussing a paper almost a decade old?
Dude, if Mann had released it at the time, we wouldn’t have had to go back almost a decade. We only had to go back because he said that to ask him for his code was “intimidation”, and refused for almost a decade to release it.
Talk about blaming the victim, man, that takes the cake. He hides his code for a decade, and then when it’s finally dragged from him kicking and screaming, you bitch about people discussing code a decade old?
That’s one for the books, Joel, that’s hilarious. I still don’t get why you are defending a serial liar, a guy who destroys evidence, but by gosh, when you do you sure provide great entertainment. Keep them excuses coming, Joel, they just get better and better.
w.
Willis,
I used the term “base code” because having read through a number of posts at climateaudit, McIntyre had to get supplementary information to make it work (apparently). Mann et al didn’t write the stuff to be user-friendly. But I’m very sure it is the source code for MBH98, as I think I can demonstrate, citing exclusively from climateaudit.
Here’s a 2008 post where Steve makes use of the code and links to it. You’ll recognize the web address to the code.
http://climateaudit.org/2008/04/05/squared-weights-in-mbh98/
And other posts where he specifically mentions using the ‘source code’ for MBH98.
Here is a post where a contributor at climateaudit transliterates the MBH98 fortran code to Matlab.
You get the picture.
Weirdly, there are few (13) posts at climateaudit on Mannian source code, and it seems interests terminates miid-2008. Steve handily provides a category in the drop down sidebar titled “MBH98”, subheading “Source Code.” You can check for yourself.
Inconvenient to our discussion, Steve never says “oh, by the way, I’ve ascertained that the code is the actual one after all”, so I guess you could dance on the head of that pin if you really wanted, but I think reasonable readers will discern that McIntyre is using the code because he believes it’s the actual one, not “as if” it’s the code.
IOW, the source code for MBH98 has been on the web since 2005; the public release occurred about 3 years after first requested by Steve McIntyre (his original request was for own use, not public release).
jimmi_the_dalek says:
April 19, 2012 at 5:32 pm
Thanks, Jimmy. There are a couple of difficulties with your example. The main one is that in climate science we almost never are doing “experiments” in the traditional sense. Instead, we are taking some dataset X and subjecting it to an often extremely complex set of mathematical transformations.
As a result, the usual ideas about “experiments” don’t hold. The first thing we need to do is to check and see if the authors actually did what they claim they did. And the only way to do that is to have access to their code and their data. Otherwise, for the 12 reasons listed above, we’re just spinning our wheels and wasting huge amounts of time. Michael Mann refused to reveal either his code or his data. Lonnie Thompson has made a living out of hiding his data. Bear in mind that those are the guys whose actions you are so vigorously defending …
Ross McKitrick’s paper on UHI is a great example of the benefits of showing your work (just like you had to in high school, just like you have to in all true science). At one point Ross made an error in the code, he used radians rather than degrees. Such an error is impossible to dig out without having the code. But because he had made the code available, the error was discovered quickly.
You are correct that doing that is a “limited check”, but it is a crucial check that should never be skipped. First of anything, before we see if we can do what they say they have done, we need to find out if they can do what they say they have done. If we find they have made some error such as the one Ross made, we can stop there and deal with that.
Or if we find they’ve actually done what they described, then and only then can we go on to do further tests or repeat the analysis in some other manner, as an independent check.
You seem to think that we should not check to see if they have made errors … I fear I don’t understand that idea at all. To use your example, suppose Michelson and Morley made some foolish error in their experimental setup … wouldn’t you want to know that before you set out to replicate their experiment? In the old days, you could look at M&M’s description of their experiment to determine that. Now, we need to look the code to determine that … and you say it’s OK to hide the code. But if M&M had hidden their actual description of their experiment, where would you be? And that is exactly what you are advocating …
In closing let me repeat, because it bears repeating, the judgement of the authors of the article, who said:
You and Joel are arguing against that. But neither of you have shown that hiding the code doesn’t carry significant costs. Neither of you have shown that hiding the code yields benefits. Neither of you have shown that hiding the code is consistent with scientific practice in other fields.
And until you can do that, I’m going with the author’s claims, because they seem self-evidently true.
w.
Willis,
just making sure you didn’t miss confirmation of MBH source code above. It would be good to resolve this bit of the conversation.
http://wattsupwiththat.com/2012/04/17/the-journal-science-free-the-code/#comment-961342
>>
jimmi_the_dalek says:
April 19, 2012 at 5:32 pm
Think experiments : you want to check the Michelson-Morley experiment – do you say it is impossible because you do not have Michelson’s interferometer? No, you build a better one.
<<
So we should build a better one. How would we decide if a design is better–or worse? How would we know without comparing our design with the original? How does that work exactly?
>>
A true check of an “experiment” which depends on a computer program does not require that you run the same calculations with the same program (and on the same machine/hardware/operating system/system libraries etc etc). Frankly that is a very limited check. A true check would insist on using an independent program with the same functionality to see if it gave the same answers.
<<
As I said, there are many naïve comments on this thread. This is another one. You can do the same thing “functionally” in different ways. Some routines are approximate. Others are more exact–but most are approximate and will give slightly different answers.
For example, if I say I’m using a function that computes the square root of a number, what EXACTLY does that mean? How many parameters are there to the routine? One? Two? Three? More? Do I return the answer as a function return, or in one of the parameters? Do I use integer arithmetic, floating point arithmetic, (or what they called fixed-point in PL/I)? (Did you know that standard floating point numbers are really a sparse representation on the number line?) Do I use only real positive numbers, real positive and negative numbers, or complex numbers? When I convert the internal module representation to logarithms (if I use logarithms) do I use complex numbers with degrees or radians? How do I handle any exceptions? Do I ignore them, stop processing, return zero, return one, return pi or e, or something else? Do I use single precision or double precision?
I could probably go on for another few paragraphs, and that’s only one function.
A program between 100 and 1,000 lines of code (a small program) would be impossible to duplicate without the original code. We haven’t even touched on the problems with chaotic systems.
>>
I am asked to referee papers in a branch of physical chemistry which often using computer generated results. Sometimes I see results which do not make sense. The authors claim they have used program X. Do I check them using program X? No of course not, that would prove nothing. I use program Y instead. If they then agree I conclude that my intuition that the results were wrong was lacking. If they disagree, I suggest to the authors that they may re-consider the methodology. That is the proper way to do a check. Insisting that they produce the code is pointless because who is going to look through a million lines of C++ or whatever. You test programs by comparing the results with another independent program.
<<
So it took ten-plus years to write program X (a million lines of C++ code). You just happen to have program Y in your back pocket? I really doubt it.
Jim
“So it took ten-plus years to write program X (a million lines of C++ code). You just happen to have program Y in your back pocket? I really doubt it.”
In my case I do – and several more programs as well. The area I work in has at least 5 or 6 major program packages that could be used to check results. The general point I am making is quite simple – demanding to see the code is probably the least efficient way to check the results. And don’t give me stuff about finite precision arithmetic – you don’t need to reproduce a result to 15 decimal places to know whether it is right or not.
What the BEST people did was much closer to the correct general approach. Yes, I know some people here don’t like their conclusions, but their approach of starting the analysis from scratch with an independent code was correct.
And Willis, I am not arguing for hiding the code – I am saying believing that demanding its release will produce benefits is over optimistic. Nor should you lump me in with Joel – if he wants to reproduce Spencer’s results he should write a program (or persuade someone else to do so) to analyse the original data streams.
barry says:
April 19, 2012 at 6:58 pm
Thanks, barry. As an indication of the incompleteness of the code that Mann provided, consider this quote from the link you provided (emphasis mine)
So in 2008, ten years after the publication of the Hockeystick, and three years after a US Congressional Committee finally forced Mann to archive the code, we still don’t know how Mann assigned the weights to the proxies … and that is central to the analysis, because all that the Mannian method does is is assign final weights to the proxies.
So once again I have to say, as far as I know we still don’t have all of the code necessary to see what Mann has done. Sure, there are a bunch of things we can learn and have learned from the code that Mann was finally forced to release. And Steve has done yeoman work to learn those things.
But we still don’t have it all.
My thanks for your perseverance,
w
jimmi_the_dalek says:
April 19, 2012 at 7:46 pm
You’re still not getting it. Suppose you check someone’s results by using another program that you have in your back pocket. Suppose your program gets different results. Both of you, of course, insist that you have the right answer.
How on earth can you decide which one is correct without examining the code?
This is particularly true in climate science, where in general there are exactly zero “major program packages that could be used to check results”. What “major program package” would you use to check Ross McKitrick’s UHI results, for example?
This is a serious question, Jimmi. If you want to get some traction for your claims, you could start by telling us what “major program package” you’d use to check Ross’s work …
If you can do that, then perhaps you could tell us what would happen if you got a different answer, and Ross refused to show his code. That’s exactly what has happened more than once in climate science. I see no way to resolve it without revealing the code.
Jimmi, Steve McIntyre and I and a host of others spent a long time trying to show that the Hockeystick contained fatal flaws. It took years to do so, because Mann (like you) didn’t see any benefit in releasing the code.
But if he had released it at the time, we would have been spared years of being beaten over the head with the Hockeystick by people claiming it was solid science.
So I know for a fact, from my own experience and my own wasted time, that there are large benefits from releasing the code—at a bare minimum, ignoring all of the other benefits people have discussed above of saving time and allowing the discovery of errors, IT DISCOURAGES PEOPLE LIKE MANN FROM LYING ABOUT WHAT THEY DID.
And if you don’t see that as a large benefit, you’re not paying attention. Perhaps whatever field you work in is not over-run with liars and crooks like climate science, perhaps in your field a host of the scientific leaders and main players are not self-confessed liars, cheats, and manipulators as they are in climate science.
Here, we need such rules just to keep the animals in line.
w.
“Jimmi, Steve McIntyre and I and a host of others spent a long time trying to show that the Hockeystick contained fatal flaws. It took years to do so, because Mann (like you) didn’t see any benefit in releasing the code.
But if he had released it at the time, we would have been spared years of being beaten over the head with the Hockeystick by people claiming it was solid science.”
So why, ten years ago, did you and other climate sceptics not have a whip-round, hire a researcher for a year to recover the original data from the various weather stations (as the BEST people eventually did last year) so that there was no doubt about the data, and then write an analysis program. Wouldn’t that have been better than spending 10 years being beaten over the head?
” Perhaps whatever field you work in is not over-run with liars and crooks like climate science, perhaps in your field a host of the scientific leaders and main players are not self-confessed liars, cheats, and manipulators as they are in climate science.”
Well yes, there is that – in my area most people appear to be reasonably honest. Perhaps the medics also have a problem, because when you look at that ‘diverse group of scientists’ that wrote this discussion paper, it turns out that the majority are in medical science/gene sequencing/protein structure research.
Jim Masterson says:
I’ve read some naive statements on this thread, but this one takes the cake.
Let’s say, for grins, that I wrote a program that thinks–it passes the Turing test with flying colors. You want to see MY code? Hey, why not just write your own?
..Well, if you wrote down for me clearly and logically how you produced the code, then yeah, it should be reproducible. If it’s not, then you either didn’t provide enough detail, or you have bugs in your code.
Call me naive if you want, but I deal with this in the real world: writing code to reproduce other people’s work, and having other people write code to reproduce mine. Our group’s code is about 5,000 lines.
W Eschenbach says:
Fallacy of the Excluded Middle. You left out … or
3) the code contains bugs, or
4) the code contains errors in logic unknown to the authors (see Michael Mann and the Hockeystick), or…
Yeah.. those are all “the original work is incorrect”. If the bugs are causing you to produce bad results, then the original work is incorrect. Same for if the authors are lying, or the code contains other errors, or the authors are shading the truth, etc etc. Or if your code is machine-specific, and so is the other person’s, then one of yours must be wrong (well, at least one).
You see that part about “… carries significant costs, yields few benefits, and is inconsistent with accepted norms …”?
Now you are here, a random anonymous internet popup, to claim there are no costs, that there are lots of benefits, and that it is in fact consistent with scientific norms … sorry, but you’ll have to do better than that.
Eh? You’re putting words in my mouth. I never said there were no costs. Of course there are costs. But I believe they’re outweighed by the benefits.
As for it being consistent with scientific norms – how many major journals require you to publish your code right now? Maybe one or two, out of dozens or hundreds?
The issue here is what economists would call “incentives”. Right now, many groups are successful because they developed something new and innovative. Of course they share their methods with other groups; that’s part of the scientific method. But being the first to develop a new code gives a group an advantage, and lets them publish new data and push forth the frontiers of science. That’s why they write the new code; to be the first to publish. It’s not for charity.
Now, say that the first time you publish anything with your new code, you have to release the code. What will you do? You might wait as long as you think you can, and publish a bunch of papers all at the same time. Or you might just not write up new code at all, if it’s not worth your time.
As with other kinds of ‘markets’, if you remove incentives for people to innovate, you’ll get less innovation.
jimmi_the_dalek says:
April 19, 2012 at 8:36 pm
I guess my writing isn’t clear. We wanted to determine if the Hockeystick was calculated correctly. We didn’t want to make yet another reconstruction, we had no interest at all in doing that. That would be just one more thing to argue about. We wanted to see if the Hockeystick was true or false, not make Anotherstick.
Because if we had taken your path, then we’d be back to dueling stories—Mann would claim he was correct, we’d claim we’d done it correctly, he’d say “you didn’t show I was wrong” … and he’d be correct. We would not have shown he was wrong.
Let me try another way to explain it.
Suppose I take dataset W and claim that I’ve subjected it to transformation X. If someone wants to falsify my claims, your suggestion is that someone who doesn’t believe me should take dataset Y and subject it to transformation Z.
Sorry, Jimmi, but that makes no sense. How would doing a different analysis of a different dataset show anything at all about the Hockeystick? Let me say it real slowly so maybe it can get across:
Science proceeds by falsification.
Now, suppose we did as you suggest, we “recover the original data from the various weather stations (as the BEST people eventually did last year) so that there was no doubt about the data, and then write an analysis program”.
How on earth would that falsify Mann’s work? You don’t seem to understand what we are about. We are engaged in falsification, which is at the core of science, and you want us to do original research instead. Certainly original research is important, but THAT’S NOT WHAT WE’RE DOING!
w.
Windchaser says:
April 19, 2012 at 9:33 pm
Or the person provided plenty of detail, but he is lying.
Or he provided enough detail but he didn’t understand what the code was doing.
Or he described the code correctly and in adequate detail but he used the wrong input.
Or he described the code in detail but didn’t realize there were incorrect constants.
So … you think the person should describe the code so well it can be exactly reproduced … but you think the person shouldn’t have to reveal the code?
What’s the difference, other than the huge chance that the English description and the computer language don’t correspond exactly?
You know, Windchaser, there’s a reason why we don’t program computers in the English we use every day … it’s too vague. If you want to know what the program is doing, you need the program itself.
w.
>>
Windchaser says:
April 19, 2012 at 9:33 pm
..Well, if you wrote down for me clearly and logically how you produced the code, then yeah, it should be reproducible.
<<
I’ve never seen a specification so complete, clear, and logical, that you could exactly reproduce a program. However, if your structured English/pseudo code reproduced all the features of your program, then you’re basically rewriting the code into wordier, structured English/pseudo code. Wouldn’t it be simpler to just provide the code?
>>
If it’s not, then you either didn’t provide enough detail, or you have bugs in your code.
<<
Or the original code has bugs not specified (there’s no such thing as bug-free code), details are missing, different assumptions are made, two programmers may read the same specification statement and program it differently, and so on.
>>
Call me naive if you want, but I deal with this in the real world: writing code to reproduce other people’s work, and having other people write code to reproduce mine. Our group’s code is about 5,000 lines.
<<
It’s interesting that your shop reverse-engineers each other’s code. Some people sure have unusual programing jobs.
>>
As with other kinds of ‘markets’, if you remove incentives for people to innovate, you’ll get less innovation.
<<
Unless you specifically put the code into the public domain, the code’s protected by copyright. If a company/university provided the funds to create it, then they hold the copyright. This is a false concern.
Jim
Windchaser says:
April 19, 2012 at 9:33 pm
You’re not paying attention. The paper itself listed four journals as examples, and I listed them again above. Why are you claiming there are only “one or two”?
And the problems you claim (below) will happen? Perhaps you could provide examples, since at a minimum four journals already have the policy. Have those journals had the problems you claim below, or is that just your claim with no evidence?
I’m sorry, but in climate science I can’t even envision that happening, where someone has code so unique that they can re-use it over and over, and thus gain a publication advantage over other researchers. I don’t even see how that would work. If you’d just give us an example of that happening in climate science, someone keeping their code secret so they can re-use it, then I would understand what it is you are talking about, and we could discuss it.
And if you can’t come up with an example of that, we can ignore it.
In any case, my friend, you’re swimming upstream. Requiring the code is going to happen across the board, no matter how hard you might cling to your outdated ideas about science and your misunderstandings about falsification, so you might as well get used to it.
w.
Willis, you ask “How on earth would that falsify Mann’s work?” to my suggestion that what you should have been doing was getting your own codes. Well to me it is very simple – you could have published it, and if Mann or anyone else objected, you would have been in a very strong position because you could have ‘shown your working’ ! At that point Mann et al would either have had to put up or shut up, short circuiting 10 years of arguments. Science progresses by falsification you say , well that is partly true , but there is more than one way to do that – instead of a direct proof that a result is wrong, you can instead produce a result that is unambiguously correct, which is preferable as it is a win-win situation. As well as getting rid of the incorrect result, you produce the correct one.
“You know, Jimmi, there’s a reason why we don’t program computers in the English we use every day. . it’s too vague. If you want to know what the program is doing, you need the program itself.”
I know, that’s why people use maths instead. I prefer to publish the maths.
@More Soylent Green! (April 19, 2012 at 9:27 am)
Simulation trash code bait is poison. It’ll make folks sick and the resultant mess won’t make an attractive story. …But it appears many folks are all excited about this, so rather than rain further on the parade, I’ll just step aside. It’s possible that a few good stories will come out of all the slogging. All the best.
Willis,
I think you’re repeating what I’ve said, but with different emphasis. Is this the source code? Yes. Does it include proxy data, weighting methods and other details? No. McIntyre is not able to perfectly replicate MBH98 because he does not have every tittle of an old paper. But researchers have not archived every tittle of their work because the research community does not replicate to advance knowledge. What we have here is a clash of cultures. Tax-style auditing borne of a political interest (MBH98 would not be an issue if it hadn’t been in the TAR), and research science, where archiving old code is much less important (because the next researcher will write their own). these details bear on the main article.
The implication I’m getting from you is that Mann et al have denied the auditors full disclosure despite having posted much of their material, whereas Spencer and co are making a genuine effort to make their code accessible, despite having produced nothing and saying it was too difficult. Now, I’ve put another slant on it here, but only to emphasise a point. There’s a way of descrbing this neutrally.
You said:
You’ve been given the link. I got it from climateaudit, where Steve describes it as MBH98 source code. He even wrote a 2005 paper on MBH, citing that as the MBH98 source code, so you seem to be at odds with Steve on that. (Yes, more information is needed)
(BTW, the 1999 paper IS the ‘hockey stick’ paper – the iconic graph in TAR)
There’s quite a bit of meta behind our talk here, but that’s not unexpected – Anthony himself credited McIntyre for the emergence of the Black Box paper in the OP (I have no idea if McIntyre is actualy mentioned in that paper). Speaking for myself, I think you’re a little strident on what is and isn’t so regarding MBH source code, which would be worth dropping the conversation for, except that the larger point about requiring code to publish is worth pursuing, and the difficulties Steve has had with what has been made available bears on that.
I am having trouble following you here, but I think you’re saying the code is trash and this concern with the code is unproductive.
Or the person provided plenty of detail, but he is lying.
Or he provided enough detail but he didn’t understand what the code was doing.
Ok, fine, you got me there. The methodology could be incorrect.
And how is this different from experiment? Experimentalists can misinterpret their results (really, it happens all the time) or lie about their methodology. You say that you just want ‘standard science’ applied, but you’re not asking to to go in and inspect lab equipment, watch the scientists perform experiments, etc.
So … you think the person should describe the code so well it can be exactly reproduced … but you think the person shouldn’t have to reveal the code?
Aye, that’s it.
What’s the difference, other than the huge chance that the English description and the computer language don’t correspond exactly?
What’s the difference between a design for an engine and having the actual engine? A lot of work; that’s what. And to build an engine, you actually have to understand the design, whereas taking one apart requires a lot less work or understanding.
In other words, before you can write your own program, you have to really understand the methodology. And that’s the kind of testing that the scientific method uses – for rigorous testing, you need other people who understand the method/concept as deeply as the first group. We don’t want everyone to just use the code without having to understand it, we want at least a few people to rigorously re-check it.. and one of the best ways to ensure this that they recreate the code, entirely from scratch.
The other ‘best way’ is to have a *very* open, well-maintained and well-documented set of open code. But that would also takes a lot of time to produce.
You know, Windchaser, there’s a reason why we don’t program computers in the English we use every day … it’s too vague. If you want to know what the program is doing, you need the program itself.
This is really not so.
Jim Masterson says:
I’ve never seen a specification so complete, clear, and logical, that you could exactly reproduce a program. However, if your structured English/pseudo code reproduced all the features of your program, then you’re basically rewriting the code into wordier, structured English/pseudo code. Wouldn’t it be simpler to just provide the code?
Code is much longer than English or math.
Here’s a rough example. Let’s say I was working on semiconductor physics in a transistor, and I said :
“I used an implicit finite-difference method to solve the electron and hole concentrations, using Gummel’s method, over a 3-point centered difference grid”.
This sentence, combined with the inputs (boundary conditions, system size, materials constants, timestep, etc), would be enough to *exactly* reproduce this work. Why? Because each of those phrases (implicit finite difference, Gummel’s Method, etc.) contains quite a large amount of information about what we do and how we do it. They’re well-established and well-defined in the literature. But the code would still take you a few weeks to write and test.
I can tell you in single line how to take a discrete Fourier transform. But how many lines of code would it take to program it?
It’s interesting that your shop reverse-engineers each other’s code. Some people sure have unusual programing jobs.
Not usually in-group replication of work, but replication of other groups’ work.
Willis Eschenbach says:
You’re not paying attention. The paper itself listed four journals as examples, and I listed them again above. Why are you claiming there are only “one or two”?
You’re right, I’d missed that. But the paper lists only three journals out of the top twenty: Science, Journal of Biological Chemistry, and the Proceedings of National Academy of Sciences. As the authors note, 3/20 is pretty far from being ‘standard practices’.
I’m sorry, but in climate science I can’t even envision that happening, where someone has code so unique that they can re-use it over and over, and thus gain a publication advantage over other researchers. I don’t even see how that would work. If you’d just give us an example of that happening in climate science, someone keeping their code secret so they can re-use it, then I would understand what it is you are talking about, and we could discuss it.
And if you can’t come up with an example of that, we can ignore it.
I’m not familiar enough with the climate model literature to say if examples exist, but I’m happy to give you examples of it from other fields. But generally, scientific simulations are dependent on inputs, so if you can produce results about one system, you can produce results about another similar system. So, that code that models one material could model another, or the code that folds one protein could fold a different one, etc. A big piece of code isn’t worth writing up if it’s only going to get you one publication.
Really, though, we’re just talking basic economics here. You’re saying that we can remove incentives for work and it won’t have any effect. Has that ever happened? I’m sorry, the burden of proof for this one is on you. History is littered with stories of societies that tried to live by “hey, let’s all just share”.
If we were only talking about climate modelling, I could see extra requirements for disclosure, as the public has a particular interest in verifying any science that turns into public policy. Basically, we need at least one of these:
1) The climatologists fully reveal their code, or
2) They stop pushing for policy changes, or
3) Their work is reproduced independently before we enact any policy changes.
In any case, my friend, you’re swimming upstream. Requiring the code is going to happen across the board, no matter how hard you might cling to your outdated ideas about science and your misunderstandings about falsification, so you might as well get used to it.
Sure, it will happen, eventually. But it’s going to face fierce resistance unless someone comes up with a way of repaying groups for the hard work of coming up with new code. If you fixed that, I’d happily get on board, but as it stands now, your interests and mine are not aligned.
Have you talked to actual, working scientists about this? I think you’d be surprised at the level of resistance. We had an invited speaker (from NIST, I think) come talk about open source a few months ago, and the questions afterward were mostly from people asking how this could possibly work, because they sure weren’t going to give their hard work over to their competitors.
PS, regarding re-use of code for multiple publications:
I have my reasons for anonymity, unrelated to this thread. But if you like, I can email you privately and share my real name, and you can see how my group has done with having our own code. I’m rather pleased: as an almost-finished grad student, I’m a co-author on papers in Nature Materials, Nature Physics, and Science. Having your own code can definitely be an advantage.