I’ve been made aware of two different opinions on state of science publishing as it relates to peer review and the pressure to publish even faster due to the Internet and all of its “instalaunch” tools.
First, in Nature, a comment by Dr. Jerome Ravetz: Sociology of science: Keep standards high.
He argues for embracing the new medium, while maintaining quality:
As more people become involved in online debates, quality need not fall by the wayside. It is encouraging to see that well-conducted discussions of points of contention between the scientific mainstream and critics are emerging, as the Berkeley Earth Surface Temperature study demonstrates (see Nature 478, 428; 2011).
Ultimately, effective quality assurance depends on trust. And science relies on trust more than most institutions. As Steven Shapin, a historian of science at Harvard University in Cambridge, Massachusetts, showed in his 1994 book A Social History of Truth, trust is achieved and maintained only by mutual respect and civility of discourse. In a digital age, civility should be extended to, and reciprocated by, the extended peer community.
Scientists have a special responsibility, but also a special difficulty. When their training has been restricted to puzzles with just one right answer, scientists may find it hard to comprehend honest error, and may condemn those who persist in apparently wrong beliefs. But amid all the uncertainties of science in the digital age, if quality assurance is to be effective, this lesson of civility will need to be learned by us all.
Dr. Judith Curry has some thoughts on this here, she writes:
I am a fan of the concept of “extended peer community” put forth by Funtowicz and Ravetz. Also, Ravetz’s phrase “the radical implications of the blogosphere” has definitely stuck in my head. Re the civility issue, I agree some level of civility is needed. Some think that Climate Etc. is too raucous (a not infrequent complaint made at collide-a-scape). A fair place for an honest debate might not be especially courteous. But the blogosphere enables a range of different types of fora and moderation rules. The challenge is to extract signal from the noise. I am pleased that sociologists are studying this.
At the same time, we have an editorial in Nature Geoscience, Embargoes on the web stating that scientists are increasingly acting as reporters now, and as a result, sometimes run afoul of publication rules. I see this as a shot across the bow against such practice.
Now that researchers, too, are acting as reporters, the guideline for talking freely to scientists but not to journalists may sound contradictory. Who should count as a member of the media for the purpose of the Nature journals’ embargo policy? The same basic rule applies: if an author actively seeks media attention before publication, we consider this a breach of our embargo policy.
At the same time, it is important to Nature Geoscience and fellow Nature journals that the scientific debate does not stop while a paper is under consideration. This principle also remains: we want our authors to present and discuss their results at conferences and communicate them to their peers. So if someone in the audience — journalist or scientist — tweets or blogs about a talk, we will not consider it to be a breach of our pre-publication embargo (see also Nature 457, 1058; 2009).
Where they say:
…if an author actively seeks media attention before publication, we consider this a breach of our embargo policy.
This squarely applies to the pre-publication publicity stunts pulled by Dr. Richard Muller and his BEST team.
People wonder why I dropped my support for him (like the feckless Dr. Peter Glieck and his science B.S. of the year awards), the answer lies within the shenanigans he pulled after earning my trust to use my data. I had always expected my data to appear in a full peer reviewed publication, instead, Muller spewed it in Congress and in his own media blitz in releasing papers that hadn’t even run the peer review gauntlet.
It may take some time (and additional train wrecks like BEST) before scientists learn that they can be their own worst enemy with these sort of behaviors.
OTOH, I’ve been considering a web 2.0 peer review experiment of my own. WUWT now has the ability to offer a peer review service for articles and papers. It is a new feature I can activate into WordPress, and would allow comments by invited reviewers to be posted for authors prior to publication, so that articles can be evaluated by a broad base of techical readers prior to publication.
I welcome readers thoughts on this idea. – Anthony
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
It would be good for science if all comments (as well as the data) were made with respect to the peer review of papers were public. Moderated blogs have an opportunity to do this. If understood correctly, Web 2.0 seems to support this.
Inspired thinking Anthony.
As many have already said; an idea whose time has come.
The peer is dead. Long live the peer.
I think it is a great idea. Bring discussion out into the open. Isn’t there some old quote, “Secrecy is the mother of corruption,” or something like that? It is only when things are kept away from fresh air and sunshine that they start to stink.
I think open discussion already is happening on this site. Of course, you now have to put up with all sorts of cracks from the peanut gallery, here. However, if peanuts like myself are not allowed to speak at a new site, may we please still lurk?
Perhaps the biggest worry would be the timid sort of scientist who doesn’t like all the slamming and banging of frank exchanges of ideas. As I recall, Darwin withheld his ideas for quite a while due to such timidness. Was it Huxley who was Darwin’s advocate? It always helps to have a brawler on your side.
As a person who has brought forth a number of incorrect ideas on this site, I didn’t mind learning where I was mistaken. “It is better to stand corrected than to fall for flattery.” In fact the WUWT peer-review seemed downright civil, especially compared to other sites. However even here it can be a bit rough, especially if one is gentle-hearted. Speaking for myself, it took me years to thicken my skin, (and I still lose it on occation.) As a teenager, at the slightest sign of critisism, and I was long gone. I imagine there are some scientists who simply wouldn’t want the fuss, and who’d rather toil in obscurity. You’d have to drag them out.
Another benefit of on-line peer review such as Anthony proposes would be the limiting of the Chump Effect, where credulous reporters uncritically accept whatever mainstream climatologists tell them regarding the latest looming disaster.
It used to be that journos would print both sides of the story. But as they say, that was then and this is now. Now we need a Web 2.0 to sift the wheat of truth from the chaff of grant-fueled globaloney propaganda. Because the media no longer does its job, and climate journals have been corrupted. Time for an end run, since the Establishment will not police itself.
Sounds like a great idea until you realize that few if any of the Climate Science establishment will have anything to do with a WUWT journal either as contributors or reviewers.
Also, open review, if that is what you have in mind, will prove largely pointless because those who volunteer comments are generally not peers in the sense that they have substantial research experience in the field to which the submitted article relates.
Several factors account for the weakness of peer review:
First, in many areas of research, there really are few peers and they may not be that bright. In that case, reviews aren’t that bright either.
Second, top scientists will often decline to review stuff by novices, so poor work often gets an fairly easy pass from reviewers as inexperienced or clueless as the author(s).
Third, in some fields, cronyism develops, where colleagues and co-workers manage to edit or review one another’s stuff.
Fourth, many scientists accept editorships mainly because it beefs up their CV, not because they intend to spend weekends and evenings reading every manuscript that crosses their desk and delving into the literature to find qualified reviewers and to enable them to make their own independent judgment of the work in hand. As a result, editors often allow or encourage cronyism, consulting only their friends and colleagues as reviewers, and then accepting the reviewers’ recommendation without question rather than doing the hard work of making their own assessment in the light of the reviewers’ comments.
A good journal needs a clever and broadly experienced editor with a commitment to science and a readiness to encourage and assist the inexperienced while, as necessary, standing up to pressure from the big names in the game. Such individuals exist and create the great journals. Anyone who’s unwilling to play that role will never edit a first rate science journal and might as well not try.
From the post;
OTOH, I’ve been considering a web 2.0 peer review experiment of my own. WUWT now has the ability to offer a peer review service for articles and papers.
There must be lots of questions.
Would it be a fully open review only process, like happens here now?
Or would it be a part closed access with “relevant” specialists encouraged to examine or replicate parts? This may be much quicker but potentially could suffer the biases and ultimately editorial mis-use that some journals have seen.
PLoS ONE is an entirely online journal offering quick review and publication that I look at often.
They are also asking questions about their own quality control and their effectiveness (and ranking?). Perhaps there are some other online examples already in existence to look at too?
The fully open process is slow and risks losing track of where it is up to.
For example I take Lord Moncktons reiteration of the “effective radiation temperature” (255K for Earth) as criticism of Nikolov and Zeller’s poster. Dr Brown debunked that by showing that the average temperature (of an atmosphere-less world with large surface temperature variations) must be much lower than the “effective radiation temperature”.
Willis Eschenbach examined that further by looking at some actual temperatures on the moon (compared to the moon “effective radiation temperature” of around 270 K) and came up with an average temperature of 196 K (in a rather sun exposed part).
So far it looks like Nikolov and Zeller integrating over the entire surface of the moon (including the colder polar regions) may be right with their average of 154 K.
If people disagree of course someone might like to replicate their result?
Willis started to look at adding atmosphere. His imagined atmosphere that is perfectly transparent to infrared will not allow the surface temperature to exceed the “effective radiation temperature”, suggesting he disagrees with Nikolov and Zeller (and Hans Jelbring as raised by Tallbloke).
I would suggest that no atmosphere is perfectly transparent to IR, and I seem to remember Nikolov and Zeller, and Hans Jelbring qualifying their atmospheres to be “thick enough” (so it intercepts a significant part of the IR going through). I better re-read them.
So, Anthony, if you can get more feedback on the atmosphere effects, then the first paper will have had its first stage review.
Or is the Nikolov and Zeller post of their poster too far back (dozens of posts and thousands of comments) for people to keep track of it.
Then how will the edited version be examined (and over how long?)
And how would it be finally “published”?
@John09
“And how would it be finally “published”?
That’s an excellent question.
Even when a paper passes peer review there will remain, in the great majority of cases, many mistakes. Thus even after acceptance in principal, a good editor will mark-up a paper pointing out errors and suggesting, as appropriate, improvements to the text, the figures, the tables.
Then, when the “final draft” is submitted for publication, a conscientious publisher will have a skilled copy editor give the thing a further work over. This may deal only with spelling, and punctuation, formatting of references, etc., or it may involve checking that what is stated in the text matches what is shown if figures and tables, checking the math, making sure that what is said here corresponds with what is said there, correcting or standardizing the format of equations, redesigning tables, and perhaps re-lettering or redrawing figures.
Today, most such work is off-shored to a low wage jurisdiction, but in that case, the work is likely done in a perfunctory way, dependent primarily on the use of software, such as grammar and spelling checkers, i.e., without input of judgement or editorial expertise. This is how the oligopolistic journal publishers now mainly do the job. There remain, however, some publishers, including the AAAS, publisher of Science Magazine, that still employ smart, conscientious in-house editors who add substantial value to what is published. But such editing in North America, cost real money.
CanSpeccy,
You seem to be forgetting the primary reason for peer review: to falsify conjectures and hypotheses. If new ideas cannot be falsified despite the best efforts of scientific peers, then they are on their way to becoming accepted theories. But science has taken a back seat to a new agenda: diverting as much public money as possible into promoting the AGW narrative.
The ossified journal review process has finally run out of steam, because it has become entirely self-serving. Scientific journals are big business, and climate journals more so, given the vast amounts of public tax money paid out annually to “study climate change”. You are right that the journal business is an oligopoly. But it is ripe for breaking:
“You can resist an invading army; you cannot resist an idea whose time has come.”
~Victor Hugo
B.Klien: I can’t get your link to work. Could you please see if it’s ok? Thanks
“You seem to be forgetting the primary reason for peer review: to falsify conjectures and hypotheses.
Not so.
The sole purpose of peer review is to determine whether the evidence presented and the analysis based upon it are valid. What that means is, assessing whether the methods are theoretically valid, the results theoretically consistent with the methods and the conclusion logically drawn.
The business of falsifying hypotheses, assuming that were the basis of scientific progress, a proposition that may be doubted, is the job of the people who write the articles that are published in peer-reviewed journals.
For example, a paper that refuted the hypothesis of AGW would be assessed by the reviewers for validity of methods and logical inference. But it would certainly not be the reviewers’ task to refute the work by making independent observations or undertaking a novel analysis, for if they were to do that, their work would constitute, in itself, a new contribution that would be deemed in need of peer review.
CanSpeccy – You make some good points about the reasons for the weakness of peer review.
The fact that “few if any of the Climate Science establishment will have anything to do with a WUWT journal either as contributors or reviewers.” and open review, if that is what you have in mind, will prove largely pointless because those who volunteer comments are generally not peers in the sense that they have substantial research experience in the field to which the submitted article relates.” I think will become of limited significance.
True Peer review has its place, but it is not the be all and end all. I think there is a place for wider review. This is because smart people are smart people regardless of their area of specialisation, and sometimes someone from the outside will quickly spot a glaring fault that those too close to a problem are blind to.
The other reason I feel inclined to disagree is because the argument for specialised peer review bear too stark a similarity to the argument used by the Catholic Church as to why the Bible should be in Latin and only the Clergy allowed to read and interpret it.
The apt response of William Tynale to that argument was
“I defie the Pope and all his lawes. If God spare my life, ere many yeares I wyl cause a boy that driveth the plough to know more of the Scripture, than he doust.”
Translation and the printing press killed that argument 500 years ago fro religion.
Personally I suspect that, given time, open review through the internet may well do the same for science.
In a way it has already started. The fact that scientific papers and reviews and discussions about them are already available on the internet means that the geni is well and truly out of the bottle and is not going back in.
dibs on name —–> WordPress SPeeR (scientific peer review)
Clever I know. Please hold the applause, I’m humble.
This can actually be a business model, at worst perhaps WordPress would be persuaded to get involved since the realm of science is large, the field focus would be required for efficiency. Categorizing, sub-categories, multiple threads within a Paper to streamline and localize discussions, minimal membership fees, application fee, non-invasive advertisments, etc. Sounds so beautiful, if only I wasn’t inept. Btw here’s another link from 11-20-2011 about this :
http://www.readwriteweb.com/archives/the_internet_will_get_a_peer_review_layer_next_yea.php
play find the faults in that article, it’s fun, and numerous.
“WUWT now has the ability to offer a peer review service for articles and papers. It is a new feature I can activate into WordPress, and would allow comments by invited reviewers to be posted for authors prior to publication, so that articles can be evaluated by a broad base of techical readers prior to publication.”
PLEASE! What would you call it? The Ultimate Acid Test? If ideas could survive review here they would be on great footing.
One point I do think you should consider. Double blind the setup. Neither the submitter nor the reviewer should know who each other is. I know most reviewers could figure out who’s paper just by looking at the evidence and seeing who was working on that but double blind is still a worthy goal.
The only way to effectively use what is in effect crowd sourcing on science is to limit the discussion to the science. Comments such as, “this paper uses the typical model used in this subject area”, or, “I recommend publishing” has no place in honest review of a paper. Identifying weaknesses, uncertainties, honesty in data representation and where possible replication of analysis, these are what help validate science.
I also believe that some journals would be willing to join in such an effort, at least as a trial. It would use this approach for the more controversial papers, providing a higher level of review prior to publication, but with the potential to move science further forward more quickly than traditional outlets.
There would still need to be an editorial board to select the papers deserving of such review and who must make the final decision on publication. Those would be journal duties and would improve the review process by not overwhelming the review community.
@Peter S
“I think there is a place for wider review”
Peter, I don’t dispute that for one moment. I think WUWT makes an important contribution by providing just that. Likewise, Steve McIntyre and others.
With luck, such review will compel science publishers to undertake their task with a greater sense of responsibility.
While a journal by a noted skeptic might attract some articles, it would be most unlikely to attract contributions from the mainstream climate science community, or be treated by them as anything much more than a propaganda effort.
I just don’t see how one could get around that problem, however great the integrity, open-mindedness and competence of the publisher.
True some people might contribute, and they might contribute interesting and worthwhile papers, but they would be branded as “the usual suspects” and it is unlikely that their work would attract funding or much mainstream climate science discussion.
It seems to me that it is best for those who perceive a warmist bias in the work of mainstream climate scientist to keep up the largely effective informal critical review.
CanSpeccy,
You’re right about the purpose of peer reviewed journals. They open the door to falsification, but their purpose is not falsification per se. But they are the self-appointed gatekeepers, and they have been compromised.
A.W. Montford made it crystal clear in The Hockey Stick Illusion that the climate journal/pal review system is heavily rigged against skeptical scientists. It is time for everyone to lay their cards on the table, and let the best hand win. Gaming the peer review system has had its day, and it is time to try something new. There will be problems, of course. But the current system is such a dishonest mess that the new imperative is to trash it and try something uncorrupted.
Anthony, an elegant solution, but possibly too simple to be valued!
I believe the correct term for this is “crowd sourcing”. Please check out
http://www.openideo.com/
Re: Journals as gatekeepers and the need for something new.
There have always been charges of reviewer bias, breach of confidentiality, and plagiarism, e.g., Peer review: the Holy Office of modern science.
So there is nothing unprecedented in the criticism that has been leveled at the review process of some Climate Science journals, although a consistency of the bias across the majority of journals maybe unprecedented.
So, in principle, I would applaud anyone rash enough to launch a journal with a declared lack of warmist bias. But on the question of starting a new journal, I admit to a prejudice. I managed peer reviewed journals for many years. It was interesting and sometimes rewarding, but it was certainly not the most relaxing vocation. So if anyone asked, my advice on launching a journal would be the same as that given by H.L. Mencken to a young man who asked advice on starting a literary magazine: “Young man,” he said “buy a revolver and shoot yourself.”
Go for it. Don’t see exactly how it gets paid for; that is one of several critical issues.
Points to consider:
1. The original has to have enough information – including new analysis of existing information – for reviewers to look at the original. Think of the slush pile at any magazine.
2. Submission must contain data, sources of data, error bars, etc. Words without math are rarely science. Central clearing house can recommend expertise for papers to get closer to completion. For climate/weather papers, this would always include professional-level statistics.
3. Need enough reviewers to be able to review any important aspects of the submission. Reviewers can work the slush pile and indicate yes/no on desire to review, with “no” given a quick reason – e.g. too busy, this paper is junk, too close to existing publications – not novel, my expertise is not in field covered by submission, math is over my head, data provided insufficient, math is over submitter’s head, obvious errors invalidate submission, no falsifiable predictions, etc.
4.. Once a submission gets a “will review” green light from enough reviewers, move discussion of pre-print to public area of web site – very much like WUWT of today. Comments should include factors omitted in paper, alternate statistical analysis to raw data, references to previous literature. Comments should minimize “me-too”, ad hominem, irrelevant/off-topic, with the option to see all. Idea is to let readers help keep signal-to-noise level high, without becoming an echo chamber.
5. At some point, need to finalize paper, with caveats and comments/discussion. That way, subsequent papers can refer to paper and comments while knowing that the previous publication is accessible and immutable.
6. Expect this to evolve into collaborative papers, as most of us don’t have all the expertise needed to do research, write using clear language, find flaws conceptual and mathematical, etc.
If done right, this could hit critical mass. If done poorly, could become theological, with different camps not communicating meaningfully.
Off topic: No poetry. I was an editor, once, and made the novice editor’s mistake of mentioning somewhere that my publication accepted poetry. Sturgeon’s Law ensued, with 90% being wildly optimistic. My revision to Heinlein’s Law (often corrupted to Hanlon’s Law). “A sufficiently advanced stupidity is indistinguishable from malice.” The converse is also true.
Thomas L.,
Thanks for the positive suggestions. Nothing will be accomplished that is easy. But it is time to sweep away the old, and embrace the new paradigm: on-line peer review.
This so much on the lines of what Anastassia Makarieva asked for in her plea for reviews here at WUWT. I’m not so sure this should not be a two step process.
One is to help through open discussions on a paper to help the authors to iron out the kinks that would surely mean a rejection if submitted for final publication without it. I have thought a couple of times in the past of publishing a paper but I would need such wide-based discussion, criticisms and suggestions. That aspect is not found anywhere today without picking a few colleagues which immediately jumps into the realm of group-think.
In such an arena, no arguments. Sorry Joel. Pointed comments and requests for clarification only. Quips, cuts and off-topics banned. But such process would have to be open to *anyone* who wants to inject pertinent thoughts, criticisms or suggestions (as my lengthy discussion with Dr. Makarieva on sailplanes and Oklahoma storms cloud action, she seemed to get valuable input from both). See, sometimes an author might get an idea to better their paper from “left field” as another commenter here just put it. If you have a process to pick ‘those in’ and ‘those out’ you automatically limit the good of such a procedure.
Two would be the formal peer review process whether status quo or on the web with limited qualified reviewers. I would so much like the cronyism to fall by the wayside. Maybe it should be a vote system of all actual scientists in the field picking the reviewers instead of the publisher, or maybe a round-robin within very specific specialties. That may not completely do away with the consensus aspect but it for sure would add some randomness and take the control and power away from a few powerful people.
We should be careful of exactly what level this process would address.
500 years ago the movable type printing press broke the control of the clergy on the dissemination of knowledge to the people of the world. But still those who would control knowledge only needed to control the press to control the dissemination of knowledge.
Today we have “the NET that covers the world”. At present anyone that wishes can communicate with any other that wishes to listen. Knowledge is now available to all and all can contribute.
Anthony has the bases and tools to create a new product Web 2.0 as a science publisher with real standing and wide spread review. I for one hope he moves forward with this concept. pg
Gary Swift says:
January 9, 2012 at 11:08 am
I would be skeptical about whether the results would be of any higher quality than current methods.
I have to agree. So long as peer review is in any way concerned with the conclusions of a paper it is bogus science. The one and only purpose of peer review should be to ensure the methods used in the study were sound.
However, that is not what we see in modern peer review. Instead we see “belief” at work, where the merits of a paper are judged by it conclusions, not its methods.
WUWT could open up a new standard in Peer Review 2.0 by allowing any and all review that speaks to the methods of the paper, and let the conclusions speak for themselves.
When relativity was introduced, many scientists of the day rejected the conclusions because they were at odds with their training and common sense view of the universe. However, they were not able to argue against the methods and over time repeated observation has shown relativity to be the most successful description of gravity available.
Imagine that in the early 1900’s a jealous scientists with a competing idea had prevented the publication of relativity, because it undermined their own work. Such an event is not uncommon in the history of science. Imagine that as a result the years of work that were required to complete the theory were never invested. Would schools still teach that gravity was a result of the flow of ether?
The debate over AGW and CO2 is no different. So long as any paper can be censored because of its conclusions, science will be held back. Peer review should only talk to the methods of a paper. Experiment and observation by someone actively seeking to overturn the theory is the true test of science. The very thing that Phil Jones did not want to see happen to his work.
CanSpeccy says:
January 9, 2012 at 6:48 pm
“You seem to be forgetting the primary reason for peer review: to falsify conjectures and hypotheses.
Peer review cannot do anything of the sort. That is the great misconceptions about Peer Review. That if a theory has been Peer Reviewed , the theory must somehow be correct. The problem starts with the scientists themselves that hold up peer review as though it meant the process in some fashion proves their pet theories correct. It doesn’t.