Guest post by Professor Robert G. Brown of Duke University.
Not all scientific research is equal, in terms of its probable impact on humanity. If one is studying poison dart frog species in tropical rain forests, getting a number wrong or arriving at an incorrect conclusion will generally have very little impact on the life of somebody living in California, India, China, or for that matter, in the tropical rain forests in question.
On the other hand, medical research has a profound impact on us all and has a long history of abuse, both deliberate and accidental. From the egregious claims of snake-oil salesmen whose very name has come to be the universal metaphor for “science” subverted to special interest to modern cases of confirmation bias and manipulation of data in e.g. drug testing, because of the potential for profit from medical science and technology it has proven to be necessary to defend the public against bad science. All medical research at this point is strongly regulated at or before the point where the rubber meets the road and actual patients might be adversely affected or killed by bad science or self-serving deliberately manipulated science.
Some of the key standards of this regulation of research include transparency and reproducibility, as well as the near-universal use of double-blind experiments to prevent the pernicious advent of confirmation bias, backed up by the threat of liability if the research process is deliberately subverted because of any sort of vested interest (including the simple desire to “be famous”, or “win tenure”, or “keep one’s grant funding”).
Engineering is a second place where doing science (in this case applied science) badly is dangerous to the public weal. If a bridge, a car, a space shuttle is designed poorly or carelessly, society ultimately pays a significant cost. In this case the needs of engineering firms and private individuals for protections of intellectual property are carefully balanced against the need to protect society. Consequently, the building of bridges, cars, and space shuttles — with or without proprietary components — is subject to oversight, inspection, and again, legal liability.
Climate research has long since passed from the realm of being a tiny discipline with a handful of researchers whose mistakes had almost no impact on humanity to being an enormous, publicly funded research machine that has a huge impact on the public weal. Whether or not you agree or disagree with the Catastrophic Anthropogenic Global Warming (CAGW) hypothesis, there is no denying that it has a huge impact on people all over the world. Quite literally every human on earth is currently at risk either way relative to the conclusions of what is still a relatively small community of scientists with a remarkably homogeneous point of view.
Whether or not these scientists are honest — do or do not sincerely believe their own conclusions is not an issue (any more than it often is in the case of medical research or large scale engineering projects); what is important to the general public is that the scientists at this point have a clear interest that potentially conflicts with their own. At risk (to the scientists) is: loss of (enormous) funding; loss of prestige; loss of political power and influence. Many of them have staked their entire reputation and career on stating conclusions as near-certain scientific fact that have a multi-trillion dollar price tag to society associated with their conclusions — however objective and well-intended — turning out to be correct or incorrect.
This dwarfs the potential damage that could be done even by unscrupulous drug companies, by medical researchers seeking to make a name for themselves, by incompetent physicians and all of the other scientific activities in the general field of medicine that are so tightly regulated. It dwarfs the damage that can be done by a faulty braking system in an automobile, by whole cities of buildings that cannot (as it turns out) withstand earthquakes, by faulty O-rings in space shuttles. It dwarfs even the damage that can be done by unregulated banking systems leading to global financial collapse that lower the standard of living “suddenly” on a worldwide scale. It is larger than the combined probable damages from all of these activities put together over any reasonable time scale.
I must emphasize that from the public point of view, this risk is in some sense symmetrical. Taking global steps such as creation of an entire cap and trade financial instrumentation in order to combat CAGW is without any doubt enormously expensive and the loss in this case is certain and immediate. However, as proponents of CAGW theories are quick to point out, society is required to make some kind of wager regardless because the cost of doing nothing if their hypothesis is true may also be extreme.
However, it must be carefully noted that one is balancing a multi-trillion dollar and immediate certain cost against a future possible cost that is by no means certain. It is simply a matter of responsible governance that the cost-benefit of this risk be soberly and, above all, openly assessed. Furthermore, both regulation and liability are absolutely necessary — indeed, long overdue — in any scientific endeavor that has long since left the ivory tower of pure research and become the basis for such far-reaching policy decisions.
Unfortunately, climate research that not only has impacted, but has led the way in the public debate and scrutiny that should correctly attend the collective expenditure of vast amounts of wealth that could otherwise be put to better use has not, thus far, been conducted in an open way. Critical data and methodology have been hidden and treated as if they were proprietary by the scientific researchers involved, in spite of the fact that the data itself has rather often come from governmental organizations or is the direct product of research funded entirely by public research grants, as in most cases is the published work itself. That this has occurred, and continues to occur, is not at issue here — the evidence that this has occurred and continues to occur is conclusive and indeed, ongoing. The simple fact of the matter is that whatever the truth of the hypothesis, the methodology and data used to support it are largely hidden, hidden well enough that it is routinely true that they cannot easily be merely reproduced by a third party, let alone the conclusions be intelligently and critically challenged.
In the ivory tower it is not unreasonable or uncommon for this sort of practice to exist, at least for a time. Scientists’ only “commodity” of value at a research University or government organization is their ideas and their research work, and theft of both is far from unknown. It is perfectly reasonable for individuals to initially hide their research goals and methods from everyone but perhaps a small set of trusted collaborators until they are proven to the satisfaction of the researcher, lest some key discovery or idea be co-opted or pre-empted by a competitor.
Even in the ivory tower, such obfuscation is supposed to — and typically does — come to an end when a work is published. Publication is the final goal of the research process in the ivory tower of the University (and often in a government laboratory) and in both cases there is a careful separation between work that is done with an eye to obtaining a patent or protected intellectual property and work being done (especially work being done with public funding) for open publication with no related rights being preserved.
Once a scientist has published in the latter case, it is expected that they will make both methods and data public upon request and invite others to reproduce and either verify or criticize the methods and any results derived from them. Anything less is a corruption of the scientific process that — when it works correctly — eventually rejects error and advances the sound. This process is often imperfect — even with levelling/protecting structures such as “tenure”, there are differential rewards to scientists based on how well they keep key ideas, methods, or even data back to maintain an advantage over their competitors and it is not uncommon for only part of the story to be told in any given publication, especially early on in the development of a new idea.
Although one can therefore understand the origins of this sort of reticence and inclination to hide research methods and data or share them only with carefully selected collaborative colleagues, and although one might even still respect this right up to the point of publication in climate research, in the specific field of climate research the public stakes are too high for this practice, however common it might or might not be in the study of poison dart frogs or the physics of graphene, to be tolerated.
The conclusions of modern climate research are almost exclusively based on published results such as the (now infamous) “Hockey Stick” graphs produced by Mann, et. al. and data sets such as HadCrut3. HadCrut3 itself is currently made readily available, but only as processed results obtained by some means from streams of raw data that are not. It is, in fact, essentially impossible for a third party to take the actual data used in the current HadCrut3 snapshot published by the Met Office at the Hadley Center, feed it to the actual code used to generate the processed data, and verify even the very limited fact that the data and the code do indeed produce the same result when run on different computers, let alone that the methodology used to produce the result from the data is robust and sound.
It must once again be emphasized that public policy decisions that have been made, are being made, and will be made in the future based on the raw data and methodology used will cost every living person on earth on average several thousand dollars, at least. Again this is stated without prejudice concerning whether or not the published temperatures are, or are not sound, or whether CAGW is, or is not, a well-supported scientific hypothesis. If it is true and we do nothing, it will cost thousand of dollars per living person and many lives over decades. If it is false and we spend money like water to prevent it anyway, it will cost thousands of dollars per living person and many lives over decades; in addition, it will do incalculable cost to the credibility of “the scientist” in the minds of the public that further amplifies this monetary damage by altering the profile of government funded research and the level of trust accorded to all scientists in the public eye.
I am writing this article to call for new legislation to address this issue, legislation that creates direct oversight for climate researchers whose work directly impacts the decision making process directing this enormous but unavoidable gamble. I am writing this as a citizen that is already paying for decisions based on the “certain” conclusion of CAGW — if this conclusion is certain, then it is certainly true that it can be transparently certain, with the entire process used to arrive at it right back to the original raw data open to public and scientific scrutiny not only by those that agree with it but by those that honestly disagree with it or merely have doubts that it is true and would like to verify it for themselves.
I would suggest that this legislation be soberly and conservatively drafted so that it in no way hinders climate researchers from carrying out their research but adds the following requirements that must both precede and follow any published result that impacts the decision process.
a) All numerical code, and input data (that is, the raw input data including any that is for any reason available from one’s source but not included in the computation, along with the provenance of all the raw data) used in arriving at some conclusion must be openly published in an immediately usable form and made readily available to anyone in the world as of the date of publication in any journal, public presentation at conferences or workshops, or publication or inclusion as a reference in policy document such as IPCC reports.
This legislation shall apply to publications based on proprietary data as well as data from public sources. The immense cost of the public decisions based on such publications and the risk of corruption of the results cherrypicked or data that might have been altered in hidden ways by vested interests is too great to permit data to be used or selected from any source that cannot be checked in its entirety, including the data that is left out.
Note well that this precise measure is indeed needed. The FOIA has already proven to be inadequate to compel the release of code and data used to generate datasets such as HadCrut3 or the “Hockey Stick” or “Spaghetti Graph” curves that currently support many of the conclusions of climate researchers.
Note also that this is hardly a burdensome requirement. It is sound practice already to carefully provide provenance and good organization for one’s raw data, to provide sound backup and revision control for the computer code used to process the data that permits “snapshotting” of the code actually used to produce a result, and to archive both for any given publication in case one’s methodology is ever called into question. The only additional requirement this imposes is to set up a website and put the data and code snapshot there with a short piece of documentation accompanying it that frankly will be of as much benefit to the researchers in the long run as it is to anyone seeking to download code and data to check results. Nowadays the cost of this is so low as to be “zero” and in any event is trivially within the means of any grant funded climate research program that almost certainly is already using one or more web servers to disseminate both results and data.
b) The establishment of a board of governance for the science with the specific and narrow purview of addressing abuses of the open scientific process. The need for such a board, and the need to staff it with people who are completely disconnected from climate or environmental research or any political organization or corporate organization with any possible interest in the outcome is clearly demonstrated in the occult Climategate conversations where it is revealed that certain researchers working in the field are far more concerned with “causes” and “winning the PR war” than with the science and are willing to deliberate tamper with data and methodology to hide results that confound their desired conclusions or to directly and deliberately subvert e.g. the objectivity and independence of the journal review process to suppress competing points of view right or wrong as they might be! The only place such discussions should occur is openly, in the literature itself, in the form of critical counter-articles or published comments, not in behind the scenes efforts to discredit editors or have them fired.
Note well that implementing provision a) will make tampering with data or methods far more difficult, but not (as evidence from medical research abuses reveals) impossible, and nothing (so far) seems to have worked to maintain any semblance of fair play in the public debate — on both sides of the climate issues. Even if the only sanction used by the oversight board is public censure and the probable elimination of future funding, those are probably enough in a world where one’s scientific reputation and ability to continue work are one’s greatest treasure.
c) The establishment of personal liability for any work that is published wherein it is later shown that the researcher did knowingly and deliberately manipulate data or methods so that their arguments lead towards a predetermined end (confirmation bias, cherrypicking data) without openly indicating what was done and why in the publications. Again, there is ample precedent for such liability (and the corresponding governance and oversight) in all scientific and technical endeavors that directly impact on the public weal, in particular in medical research. There should be considerable freedom under this rule to make honest mistakes or to pursue unpopular or popular conclusions — one of the major purposes of provision a) above is that it should ensure that there should never again be a good reason for sanctioning a researcher after the fact of publication by guaranteeing transparency — but just as would be (and historically, often has been) the case when it is determined that a published medical study where the researcher or corporation sponsoring the research “fudged” the data and as a result patients died or suffered losses makes the researchers and/or sponsors legally liable for the damage, there need to be at least limited liability and public sanctions in climate research to provide a strong disincentive to academic dishonesty or the protection of interests that, in the end, are not strictly the pursuit of scientific truth.
Needless to say, no researcher can afford to pay the true liability cost of a mistake in a ten-trillion dollar public policy decision driven by their work, but actual overt dishonesty and work performed with hidden/vested interests cannot be allowed to proceed unpunished, either. This has proven to be absolutely true in countless other, far less costly, realms of scientific, economic and sociopolitical endeavor — wherever an unregulated marginal advantage exists to be exploited by an unscrupulous individual, sooner or later such an individual shows up to exploit it. There is far too much at stake here not to protect the public good.
To conclude, much of what happens on this and other blogs, e.g. Climate Audit, is ultimately fruitless. Much energy and time is expended discussing this abuse or that abuse of good scientific methodology without any real hope of putting it right or on e.g. FOIA requests and other straightforward (but openly obstructed) attempts to simply understand how various numbers that purport to show anomalous warming were generated. The place progress has been made is primarily in a very few, but extremely significant cases (e.g. the deconstruction of the infamous “hockey stick” graph that at this point is completely discredited in spite of having dominated public discourse and public policy decisioning for over a decade) when access has been obtained to raw data and actual computational code. Mistakes that may well have misdirected hundreds of billions of dollars of public money could easily have been averted by legislation like that suggested above mandating a completely open and above-board process.
Those who advocate the CAGW hypothesis should welcome such legislature — if they have nothing to hide and their results can indeed convince “97% of scientists” as claimed, then they should make it easy for those scientists to not just read their published results (working from hidden data) but to be able to verify how their work advances from the hiddent. They, and their “cause”, can only benefit from a completely data-transparent process if their conclusion is correct.
Advocates are mistaken in treating the CAGW hypothesis as a public relations problem to be solved or a cause to be fought for in the first place (terms bandied about in a most disturbing way in the Climategate communications however they were intended), often discussed as alternatives to the far simpler option of publishing papers that address and attempt to refute competing claims, ideally acknowledging points where they might have a point. Scientific discourse has no room whatsoever for either of these as the popularity of an idea is irrelevant to its probable truth, and “causes” smack of either political or religious thinking, both of them ultimately irrational in different ways. An idea is held to be correct when it is well-supported by a mix of good fundamental science, reliable data, and openly reproducible, openly critically examined methods, and any good scientist will always bear in mind the fact that however much they “like” their own conclusions, they could be wrong.
There is little that is certain in science, and good science is honest about the uncertainties even when — perhaps especially when — there is a lot at stake.
Yes, this is a high standard of truth, one that will take time to achieve, especially in a field as complex as climatology, where many results are obtained by means of rather complex computational or statistical methods that rightly should be closely scrutinized as it is all too easy to either “lie” with or be honestly misled by an incorrect model (again this happens so often that we have a whole terminology such as “garbage in, garbage out” to describe it) or incorrect statistical analysis — the latter especially is a bete noire in sciences (and medical research) with far less impact on the public purse than climate research.
Nowhere are the stakes higher; nowhere is the oversight lower and the methodology ultimately more deliberately hidden than it currently is in climate research. And why? If CAGW is indeed true, a truly open process of research and decision making should be openly and even enthusiastically embraced by supporters of CAGW, because it will equally well compel skeptics of CAGW to provide full access to their methods and data and reveal possible vested interests. How often have we all heard the litany “anyone who criticizes CAGW is supported by the oil industry” (and seen scurrilous allegations to that effect in the ongoing discussion revealed by Climategate). Well, here is an opportunity to provide objective oversight and liability in the unlikely event that this is true — but in both directions.
I would therefore strongly suggest that a sympathetic advocate be found who would sponsor the a-b-c rules above as actual legislation to govern all climate research, publicly funded or not, that is actually used to influence large scale public policy decisions. Indeed, I would call on all climate researchers and journal editors to enforce “voluntarily” compliance with rule a) whether or not such legislation is ever written! Climategate 1 and 2 documents have clearly, and shamefully, revealed that many climate researchers currently knowingly and deliberately refuse to make either data or code/methods publicly available even when proper FOIA requests have been made. Journals such as Nature or Science have a deep responsibility to ensure transparency in any papers they choose to publish that have such a huge real cost and impact on public affairs either way their hypotheses are ultimately resolved. Papers published in climate science that specifically address the issue of global warming, including papers published in the past, should be given a reasonable opportunity to provide provenance and access to raw data and methods and, if that provenance is not forthcoming for any reason, the papers should be publicly repudiated by the journal and withdrawn.
Perhaps we could call it “Mcintyre’s Law”, since few people have fought this battle more frequently, and more fruitlessly in far too many cases, than Steve Mcintyre.
2kevin says:
December 2, 2011 at 9:24 pm
“Your argument is to maintain the status quo for essentially a right wing anarchist ideology?”
What’s your definition of “right wing,” Kev? Just asking.
This is a VERY BAD IDEA! It won’t affect the pseudoscience one bit. It will just give the fraudsters another category to spend money on, thus another way to justify increased grant proposals.
Why not consider a simpler solution, requiring no regulations, bureaucracies or taxes?
Simply write into all contracts for publicly funded science the requirement for proper use of the scientific method, including transparency and replicability.
Then, if what they do isn’t science, they don’t get paid.
Neil,
I concur that your approach is an excellent way to improve the robustness of the scientific method in Climate Science. If you don’t follow the rules (essentially A and B noted by Dr. Brown) you don’t get paid. To have some teeth the contract/grant would include a requirement that the gantee (and/or his sponsor – say Penn State for Dr. Mann) pay back all the funds received if A and B are not followed. The complience side of this could be covered by including a clause in the grant/contract noting audits are at the discretion of the grantor.
I would certainly vote for McIntyre’s Law!
As I understand it, most organisations (e.g. NASA, NOAA, scientific journals, universities) already have data transparency rules for published scientific research. The problem is that these rules have been regularly flouted by the very people whose responsibility was to uphold those rules.
If a new law could simply force these people to follow the rules properly, with severe penalties for those who did not, then that would be a big advance.
Many thanks to Professor Brown for an excellent piece. I hope his ideas will come to fruition.
Chris
Brilliant analysis, brilliant proposal.
What’s not to like?
Well, this: who is going to write such law, and how is it going to get through the various Parliaments/Senates of the countries mostly involved?
The usual political horse-trading prevalent in all our governments and those political entities who ought to crutinise governments but don’t would water this down to a toothless paper tiger.
The six-month limitation to the UK FOI is a case in point. CRU finagled these requests in such way that the Commissioner couldn’t do a thing, except state that it was a breach of the FOI law.
Also – can we wait that long until such most desirable law has been passed?
This concept brings together several of our worst professions.
1. We need laws to be written to protect the people and their property.
[ figure the odds of politicians EVER doing anything besides helping themselves get reelected. Not to mention the negative unintended consequences of every law they write]
2. The laws should be designed to hold lies and distortions illegal and not honest mistakes.
All grant money must be returned and restitution paid if the perps acted illegally.
[this would require insurance companies, another “not so popular” profession, to bond and insure the process]
3. Finally, the last profession would be the attorneys.
[need I say more?]
IMHO (and I mean very humble), it could mean maybe, kind of, sort of, possibly, a slight chance of some minor thing going wrong and the system again proves itself designed for them.
Even with all the bad possibilities I believe your proposal should be pursued.
This is a superb article and ought to appear on the front page of the New York Times, the Washington Post, The Times (London), Le Monde, La Observatore, Bloomberg, Google, Yahoo and every other media outlet ( print, broadcast and electronic ).
Bravo, Professor Brown !!!! Bravo!!!
I have reservations about regulating research in this way, although the author makes an eloquent and persuasive case. I fear the law of unintended consequences will kick in. I don’t agree that blogs like this are fruitless – at the end of the day the best weapon is publicity and these sites have become an alternative and reliable source of information that is gradually pervading the MSM.
Speaking of grant money. Briffa, in 2002, helped an outside organization ‘review’ a grant proposal.
Sylvia Herd
Senior Administrative Officer
http://www.ecowho.com/foia.php?file=1140.txt&search=research+grant+review+-+ID20020124
Taking it in, and doling it out.
The proposal sounds wonderful for the thirsting skeptics, but IMO is fraught with pitfalls. Be careful what you wish for.
In its early stages a lot of medical research is not subject to the intense scrutiny. It is only when a specific drug or therapy threatens to become commercial that the heavy legal burden becomes necessary.
Before this stage, medical research continues in its “natural” form encumbered by only the traditions of the field.
This natural distinction between theoretical and practical (commercial?) makes the implementation of an intense regimen quite a bit easier.
First, the traditions of climatology are broken and my preference would be for that part to be fixed by climatologists themselves; of course with constant nudging from McIntyre and company.
But to carry the medical (or engineering) analogy to its conclusion what is the commercial product of climate research? Policy? I am afraid the analogy breaks down and perhaps starts getting into free speech issues.
Excellent article! I strongly support provision (a), which should immediately be pressed upon all sorts of funding agencies now (i.e., it does not require legislation to begin pressing for this kind of open data as an ethical/professional expectation in the “climate science community”…..
I do have a lot of concern about (b) and (c) however, since they are more liable to various types of abuse. The open data requirement of (a) should be a “no brainer” for the reasons stated in this article and elsewhere.
However, as soon as one legislates a new board or commission it is of course open to potential influences and abuses from various directions. That is not necessarily any fatal objection to this proposal, but simply a statement that this kind thing always requires much careful study and analysis before it should be legislated.
As for (c), despite all the abuses that have been uncovered (and who knows how much more we don’t know about), an open-ended personal financial liability may be a very blunt and dangerous instrument for all concerned. At the very least it is something that would require the most careful examination and procedural protections.
But there should be no reason not to support (a) immediately both with and without legislation. I applaud calling it “McIntyre’s Law” and more generally perhaps open data sourcing should be called “McIntyre’s Principle”….. it does not need to be law in a legal sense to become a cultural and ethical norm.
Professor Brown’s article is brilliant and helpful, if we focus closely on what he recommended. He recommends legislation that requires researchers, their institutions, scientific journals, and similar entities to practice in accordance with scientific method. Specifically, he recommends that all raw data and methods must be published along with any article that relies on them. This requirement serves the Reproducibility of Results standard in scientific method and, like all elements of scientific method, it is entirely neutral with regard to the publishing scientists and their critics. The requirement would apply to disputes caused by critics who claim that some data or methods have been withheld. The requirement would foster standard methods of archiving data and methods on websites. Government would interfere with practicing scientists only to the degree necessary to ensure that the standards of archiving have been met. That is a very low level of intrusion.
If this recommendation of Professor Brown’s had been in place in 1995, the harm caused by Phil Jones, Michael Mann, and friends would not have occurred. Phil Jones never had the raw data in some reasonable form for archiving and Mann has been unwilling to address questions about his methods. (The government would not be the critic of Mann’s methods; rather, the government would require publication of all of the methods.)
Professor Brown recommends additional steps that go beyond scientific method. Creating liability laws along the lines of what is found in medical science goes beyond scientific method. That should not be surprising because pure science, such as physics, and medical science are not fully parallel. Medical science exists to serve medical practice and physicians are first and foremost caregivers, not scientists, though physicians and physician researchers use the products of pure science. If climate science is to be treated like medical science then government must act intrusively. This topic requires a separate post that I do not have time to undertake now.
In conclusion, Professor Brown’s recommendations for legislation that requires compliance with scientific method in archiving and publishing are not intrusive and do not give power to government. His recommendations about liability laws require government intrusion and must be evaluated more fully.
Wow.
“What we need is someone to stand up and say with full authority of government that the misinformation they are putting out is simply not true. THAT will shut it down.”
But it can all too easily become a cure far worse than the disease. In a sense, that’s what we are relying on now — the IPCC stands up and says with the full authority of many governments that CAGW is underway. Furthermore, imagine the consequences of such a government board in the UK at the time Mann and Jones and friends were conspiring to have de Frietas put down like a dog because he “dared” to accept a paper that challenged Mann’s hockey stick. I shudder at the mere thought of it.
A scientific review board, OTOH, staffed by scientists who are not in the game and whose only charge is to adjudicate cases where data and methods have been hidden or where unethical manipulations of that data are revealed or vested interests are revealed — that already exists, in the form of the review process of the various granting agencies and journal editorial processes. The only flaw in what we have now is that both the latter and the former are all too easily corrupted by being in the game themselves; the problem with having a small group of very powerful researchers with a homogeneous outlook is that ultimately they review each other’s papers, and review each other’s grant proposals (I am quite familiar with the process involved in both cases and this is indeed how it works). Oh, and the tenure process is more of the same.
One can all too easily get into a positive feedback “review loop” that excludes and punishes outsiders, usually to the detriment of science. This can be, and sometimes is, opposed by grant officers or editors, and generally it is benign or even sane (because in less “political” cases the consensus view is probably the right view anyway). The process needs to be, and usually is, somewhat tolerant of iconoclastic work. Climategate basically reveals that the specific case of climate research the review process has for better or worse been wrapped ever more tightly around a narrow ideological core that is highly intolerant of competing or differing views because they “don’t help the cause” or “send the wrong message” or lose “the PR war”.
To be honest, the people that bear the greatest single responsibility (and are the most at fault) in this entire debacle are the journal editors IMO they should be ashamed of themselves. They aren’t supposed to be partisan and are supposed to select reviewers in a balanced way, but Climategate makes it absolutely clear that the entire journal review process has been subverted so that none of this is true for the majority of the journals out there.
Why is de Frietas an exception that is so glaring, an exception that reveals (in the Climategate communications) the degree to which the closed community rules the journal review process? Why is it that, after M&M and work done within the hockey team itself reveals that MBH and MJ are both hogwash, pure noise that is obviously egregiously incorrect, Nature and other journals wherein these works was published or cited as key references did not publicly force the retraction of the article and the publication of a corrected version? Why is it that M&M were forced to publish their critical work in other places besides nature? The blatent “gatekeeping” that has occurred in climate research is itself a scandal.
I myself am least confident about the b) provision of all of the three. As far as c) goes one can always sue somebody, but their liability in any such suit may be strongly limited by our properly strong belief that scientific research should be free. The standard is — equally properly — altered under law in the case of the abuse of medical research for self-serving ends, and even there an oft-cited objection is that it openly encourages e.g. malpractice suits or massive suits against drug companies. Yet it was just yesterday that I read (I think on slashdot) of a meta-study of medical research that reveals that it has one of the worst track records for the reproducibility of results of any of the sciences. The review wryly observes that if one is being paid to find an effect, one often succeeds, and it isn’t until somebody comes along that doubts the conclusion and isn’t being paid that contradictory results are obtained and the effect disappears. In other words, even the loose governance of medical research isn’t enough to police the medical research community — it is all too easy to get complete crap published because (and I say this as something of an expert) nobody understands statistics!
This is the sad truth that M&M reveal. Nobody understands statistics except perhaps a select handful of professional statisticians, mathematicians, physicists and engineers. I’ve managed to master parts of the field well enough to be able to write a program (dieharder) that is devoted to pure hypothesis testing in the field of random number generation, as well as one of the world’s best (proprietary) neural network modelling tools, I can do real math well enough to teach it through the level of PDEs (I could probably manage to teach most of the courses in an undergrad math major), I’ve published a dozen or so papers in very advanced statistical modelling of critical phenomena via Monte Carlo, and I wouldn’t begin to arrogate myself by claiming that I understand statistics. It is difficult. The statistical methodology that underlies many of the currently published results in climate research is very, very difficult to understand and in some very deep sense it is being performed by rank amateurs.
That’s the group I would turn to to lead the oversight board suggested in b). The core of this group should be Real Statisticians. Not the Sears kind, the real kind. People who can look at a methodology and tell at a glance themselves if it is bullshit, or if there is a well-hidden thumb on a scale. Statisticians whose work has been in fields completely distinct from climate research and who have no record of active participation in environmental activism and are arguably strictly neutral on “green” advocacy issues. I’d throw in a physicist or two mostly because they are usually fairly sane and multitalented and a quantum physicist, field theorist, experimental worker in condensed matter physics or nanoscale physics isn’t likely to have a dog in the climate fight. The board should be staffed by people who are naturally skeptical of statistical results, who have a “show me” kind of mentality, and who are themselves iconoclastic to a fault. Skeptics, in other words, but not skeptics in the specific context of climate research.
rgb
“I would never call Mr. McIntyre’s efforts, “fruitless”, many have already borne fruit, and the seeds from those fruit will themselves bear fruit in the future.”
All too often fruitless. I was referring to M&M’s well-documented efforts, over many years, to get access to the actual hockey stick proxy data and code, and to Mcintyre’s ongoing effort to gain access to still more actual code and data in the case of many other questionable publications. These efforts are so far fruitless, and they each require enormous energy to pursue — I literally don’t know how he does it, as he has to fight for years to get something that he shouldn’t even have to ask for.
That’s the fundamental motivation for Mcintyre’s Law. He shouldn’t even have to ask for access to the code and data and methods used to produce graphs and conclusions that affect the directed use of trillions of dollars wordwide! Indeed, this is so obvious that it is manifestly insane that he has to actively fight against researchers who don’t want to expose their work at a level that might reveal the hidden thumb on the scales. I can understand it — that thumb could be deliberate or it could be sheer incompetence in statistics (nobody understands statistics, making it so very easy to either lie with it or simply make a horrible mistake with it) but either way the discovery of a serious problem could be a career-ender. However, the stakes are too high for the current business-as-usual hidden methodology or methodology described only in crude terms that do not permit the precise numerical reconstruction of results to continue.
rgb
StudioBronze says:
December 2, 2011 at 7:22 pm
It will be very difficult to make a compelling counter-argument to Professor Brown’s proposal that is not transparently self-serving. Good Job.
____________________________________________
NEVER make the mistake of thinking a politician can not twist a good idea into something that serves his masters, the large corporations and the banks who fund his campaigns.
So actually it is not hard to make a compelling counter-argument. All one has to do is look at what has happened to the US Food Safety Laws and the FDA and USDA. Laws are only as good as the law makers, regulation makers and the BureacRATS that enforce them. I call Washington the “District of Criminals” for a darn good reason, six years of digging in the stinking muck DC generates.
Like this proposed law, the food and drug laws started out as a “Good Cause” The food and drug laws have since morphed into a monster used by “self-serving” interests to control the market, wipe out competition and in the last few years, to actually make people ill as part of a campaign of Problem-Reaction-Solution. The goal is becoming the readily apparent Control of the food supply. see Farmland grab: http://farmlandgrab.org/
Corporate/Government Revolving Door:
http://www.rense.com/general33/fd.htm
http://www.cptech.org/ip/health/politics/revolvingdoor.html
Creating the PROBLEM
After WTO agreement on Ag was ratified, the USDA/FDA moved to get rid of the “Safest Food in the World” by changing to the international HACCP regs.
1996 HACCP http://mfu.org/node/276
By opening the doors to uninspected imports (WTO) and removing hands on government inspection of food processing plants (HACCP) there was a doubling of food borne illness. (My analysis of CDC statistics after graph was taken off internet)
The Media of course played this up and instead of identifying the WTO agreement on Ag AND the new HACCP regs, the “Old laws” were blamed and a “New law” was waved on high as the “Solution”
“PROBLEM”:
Shielding the giants: https://spideroak.com/directdownload?platform=ubuntulucid&arch=i386
FDA says it is OK to turn bad food into sellable stuff: http://www.peoplesworld.org/fda-says-it-s-ok-to-turn-bad-food-into-sellable-stuff/
“SOLUTION”
The “Solution” is a law that turns control of the US food supply over to the Ag Cartel who wrote the World Trade Organizations Agreement on Ag. (Dan Amstutz – VP Cargill & Goldman Sachs employee) The New Law literally names the World Trade Organization as a must follow.
Trojan Horse Food Safety Law: http://www.examiner.com/scotus-in-washington-dc/trojan-horse-law-the-food-safety-modernization-act-of-2009
The reaction of farmers? “Let them eat GRASS!” http://www.newswithviews.com/Hannes/doreen110.htm
Cartoons speak farmers contempt of the USDA even better: http://www.naisstinks.com/index.php?con=cartoons
How Goldman Sachs created the 2008 Food Crisis http://www.foreignpolicy.com/articles/2011/04/27/how_goldman_sachs_created_the_food_crisis?page=0,1
“The problem is that, because government gets all its funding by coercion, it is outside society’s matrix of exchange. This means that, in providing a service, it has no rational way (rational in terms of the evaluations of the intended consumers of the service), to know
a) which service to provide, and
b) whether it is providing too much, not enough, or just the right amount.
….
What if those assumptions are false?”
Well, you could try looking at the evidence, instead of spouting ideology… no, I suppose not.
I won’t even bother to go down the list of things that are features of your daily life that would not exist if it were not for the government funding of research. There are, perhaps, places where the government spends tax monies unwisely, and I would certainly not want to interfere with your constitutional right to speak on any of them including government funded support of technological and scientific research.
I might, however, gently point out that doing so by typing a reply into a computer that uses a network built on top of software nearly every single part of which has roots that can be directly traced back to government funded research and corporate research performed by people who were educated in University programs that were funded by the government and that is carried from point to point “for free” (for the low cost enjoyed by end users) because of government subsidy of the networks themselves is a bit hypocritical.
From the Enlightenment on, the majority of scientific discoveries and advances have been made by virtue of taking our very brightest people and providing them with some means of living that doesn’t involve digging ditches. The original discoveries were usually made by the only class with leisure time — the nobility — or those directly supported by the nobility to perform such research, all funded by money removed by “coercion” from the commoners. However much you may have bought into the mythology of Edisonian research and Atlas Shrugged (one of my own favorite books growing up — note well that I was a physics and philosophy major as an undergraduate) it is well worth remembering that Ayn Rand herself accepted medicare and social security to help pay for her terminal cancer (caused by her two pack a day drug addiction).
Reality isn’t ideal, and a whole lot of what government pays for — such as the satellites that actually provide sound data that is in the process of gradually helping us to understand the physics of the sun and the science underlying both climate and weather, satellites that have paid for themselves thousands of times over in increased crop yields, decreased damage and mortality in hurricanes, and the countless minor regulations of everyday life that are optimized by e.g. knowing when a frost is coming, being able to postpone a trip due to anticipated rain. The evidence, in other words, that civilization benefits from government works funded by “coercion” is so vast as to be (in my opinion) conclusive.
Is all of that money perfectly spent? Of course not. Neither would it be perfectly spent in any other scenario. Perfection isn’t on the agenda today, as it, along with idealization, are both mythologies, attractive illusions. There is plenty of freedom in the current system for Edisonian work to succeed. There is enormous freedom for corporate funded research to succeed in the best of capitalistic traditions, with significant protections for intellectual properties developed by such work (too much protection in many cases, as most of it is based on work that is already in the public domain, paid for by government money and published so that anybody can use it, creating a whole new form of “free enterprise” known as “the patent troll”). But the heart of the engine is indeed Your Tax Dollars At Work. Shut them down and you will bring on the greatest economic depression the world has ever seen, as the only thing that is keeping our entire world civilization out in front of the growth in population is our science and technology. As the world’s physical frontiers are largely eliminated so that there is less and less low-hanging fruit to be plucked, progress along the scientific frontiers has been picking up the slack, and only the government or corporations so large that they are themselves “governments” can afford to support the entire infrastructure that produces scientists and new scientific understanding that ultimately fuels the engine.
Ayn Rand, alas, got it wrong. Not completely wrong, of course. It’s just the the optimal point for a rational society is somewhere between the poles of Ms. Rand and Mr. Marx. Both the endpoints are clearly unsuitable in well-known ways. In between, finding the optimum is a difficult problem in multivariate optimization on a time varying surface, it is surfing the wave as the wave itself changes. But we digress…
rgb
Well written, very clear and reasonable. However, I am not sure that I want to see the government regulating any more than it already regulates, nor do I want to see more laws prescribing human behavior, especially regarding research. The government screws up everything it touches.
The problem is that scientists themselves failed at their job. Not enough people spoke up and demanded the appropriate peer review as they should have. Funding needs to be reformed, and scientists need to police themselves using the methods that have always been in place. What needed to happen didn’t happen because scientists were afraid to challenge a small group of men who had somehow managed to co-opt and control the direction of scientific inquiry in the area of climate.
It all happened with the blessings of government, the IPCC and private funding sources who were only to too happy to throw millions (billions?) at an idea that became a cash cow and provided a political advantage.
Scientists of good conscience need to stand now, speak as one voice and demand the reasoned inquiry you propose. But, government rules? I’m not so sure about that.
http://avalon.law.yale.edu/20th_century/eisenhower001.asp
Eisenhower redux, except for the call for more laws and more government. What board could be appointed by the current USA administration that would honestly assess anything?
That said, everyone on this globe should read this article.
@Robert Brown
Thanks for your wonderful article and elaborations. Your comments about the issues with the statistics involved remind me of one of the recurrent questions I’ve had when reading about the real intellectual scandals associated with CAGW:
why can’t we get more involvement from real, serious statisticians and mathematicians to review the quality (or lack thereof) of the work of The Team?
I know some of the obvious reasons, such as (1) real math whizzes often don’t like messy issues with “politics” involved etc.,and (2) real math whizzes can’t be bothered even looking at a lot of the slovenly output of The Team, etc.
But still, with the intellectual standing and integrity of statistics and mathematics often drawn into these shoddy practices, I would think there ought to be some real statisticians out there who would want to take a serious look at this…..
If nothing else to draw clearer lines about what kinds of applications and practices are and are not justified…..
These issues have been discussed forever by scientists and others all over the world. Would Dr Brown suggest we have a grand judge, a super monitor of methods, findings and supervising and adjudicating the work of thousands of scientists who presently work independently. I’m sure they prize this independence and their work requires the freedom to think and act without censorship in any way. You can see how damaging this monitoring approach worked during the Bush administration.
The research speaks for itself, and the professional standards required in each field set limits for all research endeavors and subsequent publications. While there will be many who disagree with methods employed or the conclusions drawn by individual researchers, the net effect of the collective work of the world’s scientists, all published in peer reviewed journals, results in an advancement of our knowledge.
This is not a PR exercise, or some kind of malicious game being played out by conspirators with a vested interest in making dramatic change. Those who inpute these motives, are simply wrong.
We have reached a point in our discussion where it is no longer acceptable to proceed without an honest acknowledgment that the research in climate science clearly focuses on varieties of problems, which require our (the world’s) immediate attention. Unbiased risk assessments conducted by many, all over the world, have arrived at this conclusion.
Should it not be called the Mann Act 2.0… Prohibiting the transportation of prostituted science across state lines for immoral purposes?
As a physicist, engineer, and physician, the points made by Prof. Brown resonate strongly with me. I have made and are similar points elsewhere in the climate discussion. The rigor that is enforced on medical research comes from hard won lessons in all the ways research can go wrong. A big part of medical training today is to learn the dozens of subtle tricks that can be played by and on data, either intentionally or not – usually not. A physician with a solid background in evidence-based medicine is well-qualified to tease out methodological flaws in all kinds of scientific publications.
The current generation of climate researchers started their training in a world where their results were of mostly academic interest. Wrong results couldn’t really hurt anyone. But they had academic careers that flowered at a time when their results were used to reallocate resources on the scale of trillions of dollars. This parallels the kind of events that took place in medicine a few decades ago, when a large, multinational pharmaceutical company could partner with university biochemistry lab working on a medically promising drug. They could subtly steer the research results to show the drug in the best possible light, sometimes better than was true, all with the noble intention of bring to market a blockbuster that would help millions of people live longer, healthier lives. That, and also making tons and tons of money.
I certainly don’t think the problems are rooted in climate researchers being bad or stupid people. Like Orson Scott Card’s Ender Wiggin, they woke up one day to be told by their political leadership that the video games they have been playing might actually be used to “Save the Planet (TM),” and that vast resources will now be at their disposal. Surely this is a heady experience like none other.
The medical field has learned its lesson, and continues to get the occasional refresher course. The methods, results, and conclusions we now see in many climate publications wouldn’t pass the sniff test in a major medical journal. The climate research field hasn’t matured in the same way. I think it is probably at least 20 years away from growing up to the point where medicine is today.
If I invent a new blood pressure drug to potentially be used by hundreds of millions of people, the questions asked by the medical system are, “Is it safe?”, “Is it effective?”, “Is it better than the alternatives?” If CO2 reduction policies were evaluated in these terms, with the methodological rigor of a new drug, they would go nowhere. If I objected to evaluating my drug rigorously, and appealed to the “precautionary principle” that in the case that my drug works, millions of cases of stroke, heart attack, and kidney failure might be prevented, so it should be mass-distributed, I should be laughed out of the medical field.
Talk about lack of transparency …. I was looking at the old harryreadme file and found this:
“After an email enquiry from Wladimir J. Alonso (alonsow@mail.nih.gov), in which unusual behaviour of CRU TS 2.10
Vapour Pressure data was observed, I discovered that some of the Wet Days and Vepour Pressure datasets had been
swapped!!
The files I was looking at were decadal, 1981-1990.
Vapour Pressure, January: Min 0 Max 310
Vapour Pressure, February: Min 0 Max 280
Wet Days, January: Min 0 Max 3220
Wet days, February: Min 0 Max 3240
So I wrote crutsstats.for, whioch returns monthly and annual minima, maxima and means for any gridded output file.
Tried it on the full runs, and they look OK:”
——————-
So, I sent an email to Wladimir J. Alonso (alonsow@mail.nih.gov) and asked him…………
“Sir,
Who is Harry and how are you working together?
I found this in the harry read me file.”
“That’s why I profoundly disagree with this statement: “To conclude, much of what happens on this and other blogs, e.g. Climate Audit, is ultimately fruitless.” ”
See previous reply (I’ll say this here once and for all, as this is a common theme). Of course it isn’t completely fruitless. However, blog comments, no matter how thoughtful, usually have relatively little impact on public policy. The readers of a blog are largely self-selected according to prior beliefs and conclusions. How successful are they at changing people’s beliefs? Somewhat, to be sure, but as I think was pointed out yesterday in another thread, it is akin to arguing with devout theists about the age of the Universe (something I’ve spent hundreds of thousands if not millions of lines of prose text doing in other contexts). “Pissing into the wind” does not begin to describe it.
What has made a major impact is the published work of M&M. Indeed, I would say that it has had the greatest impact of any single thing in the entire debate. I started becoming a “skeptic” some nine or ten years ago, largely after doing some personal research on the connection between solar state and global temperature and discovering http://www.john-daly.com/#cru quite by chance. Daly’s site led me to the M&M paper, which I read and even though I’d never even heard of the “hockey stick” or given any real thought to the CAGW hypothesis (which was still far from dominating public discourse at that point) their work was instantly convincing. As a scientist, all I could think of is “who are these turkeys, what is this all about, and how are they getting away with this?”
The answer to the latter became clear over the years in between, of course. At stake was vast amounts of money, spent worldwide — more than enough to attract the greedy. They were getting away with it because they used every trick in a rather corrupt book — cherrypicking data (and bragging about it), confirmation bias, gatekeeping in the granting agencies and journals, the use of “public relations tools” backed by celebrities working for “the cause” instead of open debate in scientific journals even when one of their own number would suggest that maybe they should participate in the latter! It became clear that “climate science” was indeed an unwritten conspiracy between three groups — environmentalists and deep socialists with one agenda, a very shadowy group of money and bank and political people seeking to make an enormous amount of money and accrue a vast amount of political power by controlling an entirely artificial globe-spanning commodity, “carbon” (note the familiar theme of many blog posts, “follow the money”), and an amazingly small group of climate researchers who were literally seduced by the IPCC.
The extent of the seduction is clearly visible in one of Phil Jones climate reconstructions from immediately pre-IPCC, and from some of the earliest comments from climategate 1, where people like Briffa were not yet firmly on board and were wondering who this Mann kid was. Jones published a graph (I’m too lazy to go track it down and don’t know how to embed graphs properly in this interface yet) that clearly shows — I think even labels — the MWP and LIA, and shows a warming now that is roughly commensurate with that of the MWP. Certainly no “smoking gun”, no need to panic, no “catastrophic” process evident.
At this point his work was, I’m sure, still unbiased as nobody cared. I believe some of Briffa’s work at this stage was similar — even Hansen’s reconstructions showed a clear MWP and LIA still — but again I’m too lazy to go back and look up papers so take this with a grain of salt.
Both of them were virtually ignored in the IPCC process, while this unknown upstart, Mann, was positively lionized. They had proof at last! A smoking gun! Catastrophe evident (even people who flunked graph reading 101 in high school could follow the end of the hockey stick up to disaster).
MBH became the poster child of the CAGW scam, funds flowed, and it rapidly became “join us or die” for anybody working in the field. Where the “death” involved is to be ignored, your work unfunded, to miss out on being lionized yourself, being interviewed on television and radio shows — for scientists who up to then were literally nobodies in the sense that perhaps one or two dozen people in a single, very select research community might have known their name and their work, this was heady, seductive stuff! Paleoclimatology and climate reconstructions went from being something nobody cared about to the most important thing in the world, to being the key to saving the world, if and only if one thing was true! If the temperature of the present is indeed “anomalous”, unexpected, impossible to explain by any other means than the increase in CO_2 concentration, then paleoclimatology is of world-shaking importance. Otherwise, it goes back to being interesting, useful, of some value — just like studies of poison dart frogs or the evolution of dinosaurs, but no longer a trillion dollar save-the-world holy mission.
Although I do blame them for being seduced even by the chance to be Luke Skywalker fighting the Darth Vaders of the Oil Empire, I also pity them. They are hardly the first good people to be unable to resist the siren song of a world-shaking ideological bandwagon, and the human mind is sufficiently subtle that it is perfectly capable of convincing itself that something incorrect is true once a certain level of commitment is made, and often our tendency to do this is made stronger by weak or marginal evidence. Hence “High voltage power lines cause cancer”, and homeopathic dilution (water “memory”). Hence belief that advanced beings will come to earth in UFOs and take a select group of the “saved” away.
The phenomenon psychologists study that is relevant here is called “cognitive dissonance”, and there are numerous case studies of the ways humans are capable of completely reworking reality itself (which is, recall, a mental phenomenon to each of us) to lie in accord with any sufficiently strong belief, where (paradoxically) evidence that contradicts that belief makes our belief even stronger.
The siren song of cognitive dissonance is all too apparent on the page of this blog as well — it is all too easy to oppose the CAGW hypothesis to the extent that you ignore evidence that supports it, or even twist it around into evidence that it isn’t true. I’m very careful to state that I doubt the CAGW hypothesis, not that it is or isn’t true. I have a rational basis for my doubt, not the least of which is a rational basis for doubting the objectivity of much of the supporting evidence. The law above, you will note, doesn’t address the truth of falsity of the hypothesis in any way — it addresses the objectivity of the evidence, because I am certain that the evidence has been produced and manipulated in non-objective ways for non-objective ends. Climategate communications are absolute proof of this, especially in tandem with M&M.
Ultimately, if you remove the factors that inappropriately protect the status quo — hidden methodology and goals, lack of an open critical process, gatekeeping in the journals — the science will fairly rapidly correct itself. Blogs are absolutely useful in the process of calling for changes in the law and the process itself. Indeed, it would be interesting to see what RealClimate would say in response to this modest proposal. I’m reasonably certain that they will either claim that the research is already transparent (in which case why would they oppose legislating this trasparency?) or claim that they have the “right” to conceal their methodology and data outside of what they publish. But I could be wrong. Perhaps they’ll respond to this proposal without my posting there (as I’m uninterested in posting there, given its origins and theme) but it is a lot more likely that they will just try to ignore it. But in the end, blog posts are not refereed research papers published in Nature, and to really change the perception of the science by means that are different from “winning a PR war” is to do more science, and correct the science that has been done, and publish it in the journals.
To conclude: Blogs are not useless, but publishing things in a blog is not equivalent to publishing something in nature. Blogs are important in the “PR war” to be sure, but the very fact that such a thing is necessary is a sad, sad commentary on the state of the science and its corruption by the political process to which it is tied. They also have a strictly limited effect on the public mind, on the attention of the media that ultimately directs widespread human belief. At this point there is gatekeeping that extends well into the supposedly objective media, but the media are notorious for following their own interest, which is often being first to break a juicy scandal. I’m very interested to see if Climategate 2 eventually percolates into the mainstream media. It would be very interesting to see an e.g. “60 minutes” devoted to MBH, M&M, and the climategate revelations…
rgb