I’ve been made aware of two different opinions on state of science publishing as it relates to peer review and the pressure to publish even faster due to the Internet and all of its “instalaunch” tools.
First, in Nature, a comment by Dr. Jerome Ravetz: Sociology of science: Keep standards high.
He argues for embracing the new medium, while maintaining quality:
As more people become involved in online debates, quality need not fall by the wayside. It is encouraging to see that well-conducted discussions of points of contention between the scientific mainstream and critics are emerging, as the Berkeley Earth Surface Temperature study demonstrates (see Nature 478, 428; 2011).
Ultimately, effective quality assurance depends on trust. And science relies on trust more than most institutions. As Steven Shapin, a historian of science at Harvard University in Cambridge, Massachusetts, showed in his 1994 book A Social History of Truth, trust is achieved and maintained only by mutual respect and civility of discourse. In a digital age, civility should be extended to, and reciprocated by, the extended peer community.
Scientists have a special responsibility, but also a special difficulty. When their training has been restricted to puzzles with just one right answer, scientists may find it hard to comprehend honest error, and may condemn those who persist in apparently wrong beliefs. But amid all the uncertainties of science in the digital age, if quality assurance is to be effective, this lesson of civility will need to be learned by us all.
Dr. Judith Curry has some thoughts on this here, she writes:
I am a fan of the concept of “extended peer community” put forth by Funtowicz and Ravetz. Also, Ravetz’s phrase “the radical implications of the blogosphere” has definitely stuck in my head. Re the civility issue, I agree some level of civility is needed. Some think that Climate Etc. is too raucous (a not infrequent complaint made at collide-a-scape). A fair place for an honest debate might not be especially courteous. But the blogosphere enables a range of different types of fora and moderation rules. The challenge is to extract signal from the noise. I am pleased that sociologists are studying this.
At the same time, we have an editorial in Nature Geoscience, Embargoes on the web stating that scientists are increasingly acting as reporters now, and as a result, sometimes run afoul of publication rules. I see this as a shot across the bow against such practice.
Now that researchers, too, are acting as reporters, the guideline for talking freely to scientists but not to journalists may sound contradictory. Who should count as a member of the media for the purpose of the Nature journals’ embargo policy? The same basic rule applies: if an author actively seeks media attention before publication, we consider this a breach of our embargo policy.
At the same time, it is important to Nature Geoscience and fellow Nature journals that the scientific debate does not stop while a paper is under consideration. This principle also remains: we want our authors to present and discuss their results at conferences and communicate them to their peers. So if someone in the audience — journalist or scientist — tweets or blogs about a talk, we will not consider it to be a breach of our pre-publication embargo (see also Nature 457, 1058; 2009).
Where they say:
…if an author actively seeks media attention before publication, we consider this a breach of our embargo policy.
This squarely applies to the pre-publication publicity stunts pulled by Dr. Richard Muller and his BEST team.
People wonder why I dropped my support for him (like the feckless Dr. Peter Glieck and his science B.S. of the year awards), the answer lies within the shenanigans he pulled after earning my trust to use my data. I had always expected my data to appear in a full peer reviewed publication, instead, Muller spewed it in Congress and in his own media blitz in releasing papers that hadn’t even run the peer review gauntlet.
It may take some time (and additional train wrecks like BEST) before scientists learn that they can be their own worst enemy with these sort of behaviors.
OTOH, I’ve been considering a web 2.0 peer review experiment of my own. WUWT now has the ability to offer a peer review service for articles and papers. It is a new feature I can activate into WordPress, and would allow comments by invited reviewers to be posted for authors prior to publication, so that articles can be evaluated by a broad base of techical readers prior to publication.
I welcome readers thoughts on this idea. – Anthony
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
I’m not really comfortable with peer review by like minded souls. Its my competitors and rivals who are my harshest critics.
If there is a weakness or a fault, they’re right on it, no holds barred.
If you want supportive views, ask your Wife, Friends, Co-researchers.
If you want a scales-off perspective, ask your Mother-in-law or Teenage child.
“We did away with the paper work years ago………”
ok, well one other thing…..
kept open and honest…..the internet can rip a new butt hole in most of the garbage that’s being rubber stamped today……….
A journey of a thousand miles begins with a single step.
An excellent idea whose time has come. Public discusssion has the potential to greatly further the search for answers. In any case, the old paradigm is in failing health … keeping it on life support will help no one.
I love the idea. Though quite frankly, if I write a paper, I don’t care if the person that stops me from stepping on myself in public has a PhD or not. Someone who just finished undergraduate statistics might actually be the one to find that bias that crept into my analysis. I’d rather be right than dignified in the end.
Leon Brozyna says:
January 9, 2012 at 12:05 pm
I read that originally as “A Journal of…” they say reading is fundamental, but perhaps that is what we are seeing: the birth of a new Journal online.
My suggestion is to have a broad range of reviewers since “climate science” touches on so many disciplines. Research that uses tree-rings ought to be looked at by botanists, for example, just to catch unfounded assumptions by non-biologists and identify weaknesses in an hypothesis. The group obviously will include technically competent reviewers, but shouldn’t be limited to them. Communications specialists as well as mathematicians are needed.
I think I am missing something, like how it will work.
1. Does each reviewer get one vote? (If so, why is it not a popularity contest?)
2. How long does the review process last?
3. What will the review process entail?
4. With each specific criticism, what constitutes adequate correction? Agreement by just that reviewer and the author?
5. What is the method of formal criticism?
6. The whole process could easily become unmanageable discussion or argument.
7. Who decides what the final form shall be?
8. Will editors/referees/moderators still be necessary? If so, how chosen?
9. Will any minimum standards be applied?
10. I assume the structure will be posted on a link.
11. Will accepted papers be listed with such central clearing houses as Springer or Wiley or Mendelay?
12. Will papers be behind a paywall? (Nothing is stupider than 80-year-old papers that some journal expects one to pay $30 for.)
13. How is at all paid for? Reviewers pro bono still? Editors? Ads? (Ads wouldn’t be a bad idea.)
14. Pre-publication media hype – what effect would it bring down upon the head of the researcher?
15. The biggest one maybe: Will secrecy be done away with? (In all the years of the existing peer-review process, what was really behind keeping the reviewers’ identities secret? A: Shouldn’t they have been given credit? B: Didn’t the researcher have the right to face his accusers?) With open-peer-review (OPR), all the cards are on the table. It seems like a much better system.
Certainly that list does not exhaust the pertinent points to address.
All that said, there is no one forcing any researcher to submit to any one journal. This means more or less that the journals that currently exist more or less just declared themselves to be a journal, then sat back and waited to see if anyone would submit papers.
Bottom line? There is no harm in trying.
I agree – Go for it!
I am wondering if anyone can assist me. I am trying to find historical weather records for Winnipeg and Regina but Environment Canada seems to have lost the data. It happened about 2 years ago, when they said at the time that they were revising the data.
@Stephen Fisher Rasey
I hope you mean Gutenberg. (Zu) Guttenberg is a synonym for blatant and unrepentant scientific plagiarism in Germany.
@Gary Young 12:23 pm:
“My suggestion is to have a broad range of reviewers since “climate science” touches on so many disciplines. Research that uses tree-rings ought to be looked at by botanists, for example, just to catch unfounded assumptions by non-biologists and identify weaknesses in an hypothesis.”
Why not open it up to whomever has any input at all?
In the beginning of The Royal Society, it was a meeting in a real room/hall, and the members would bring their experiments and run them, right in front of everybody – biologists, geologists, astronomers, physicists, etc. – and everyone had a free shot at the presenter.
Isn’t Web2.0 (which includes blogging, but would also include this set up) essentially the same thing as an open forum? “Forum” means both the PLACE of a meeting or interaction on one hand, and the open interchange itself. “Forum” implies both a level of decorum AND a level of chaos. There has been too much decorum in Journals, so much that they have become ossified, too crystallized. It is time for a new form, an open forum, just as Anthony is considering.
Anthony, I would propose that the name include the word “Forum.” As in “The Science Forum” or some such.
As the BEST SCIENCE BLOG OF 2011, and a perennial candidate, WUWT has at least a tiny standing from which to build.
From the scientists point of view, it will not get much recognition unless you can convince someone to include it in their index. For topics related to living systems PubMed might work, but something else might be needed for purely physical/chemical studies.
Nothing new here, but I do want ot point out that Nature and Scientific American have been a major part of the problem with both of them not only promoting incivility and intolerance, but junk science.
Go for it, Anthony.
You built WUWT as a civic model providing open discourse about “climate science” among the interested — in the interest of mankind, which ultimately pays as science and policy seemingly are led astray together. WUWT has brought together scientists and interested non-scientists in a civil, well-informed discourse regarding whether the Emperor is wearing clothes, or not.
I trust that you will bring the same civility, transparency, and honest heart to the attempted upgrade of a clearly broken peer review process.
I am in quality assurance and I can tell you it has nothing to do with trust and everything to do with verification. When I get a shipment of parts in, the first thing I do is open the box and verify it is what was ordered – that’s qualtiy assurance. You get into trouble when you ‘trust’ and not check at least once in a while.
Peer review should work the same way. Data and methodologies should be provided so that the reviewers have enough information to verify what is claimed – plain and simple.
Anthony,
I think your web 2.0 project is a great idea. Go for it.
Anthony, “a Web Peer Review”. What a Great Idea! The old system is broken and was hijacked by those in it for the money.
You web 2.0 idea will be done by someone. You. yourself need to do it now!
From Dr Ravetz: “Ultimately, effective quality assurance depends on trust. And science relies on trust more than most institutions. ”
—————————————————–
This makes no sense to me. The scientific method relies on replication of results, not trust. Just as the scientific method includes skepticism of claims, not trust in the scientist making claims.
I smell a Ravetz!
From wikipedia : “Post-Normal Science is a concept developed by Silvio Funtowicz and Jerome Ravetz”
** Rant on **
If science was like trading in equities then maybe Ravetz would have a leg to stand on. But I see science as an epistemology, a way of knowing. The only thing that makes science internally valid as an epistemology is the ability to replicate the work! If you can’t replicate the work, then it might be “social science” (p < 0.05) but it isn't science.
** Rant off **
With my rant now over, I fully encourage you to go full steam ahead on the web-based peer review project. Based on all of the unofficial volunteer reviewing that is performed on your fine website it seems there already is a willing pool of reviewers ready to go at it.
There are a lot of mistaken impressions about peer-reviewed journals. The peer review is just an editorial quality check, but the reviewers don’t try to re-create the graphs, or check the math or the methodology, they just check the text. The real quality check is when other people try to replicate the results.
I agree with ‘Height of Nonsense’ (Tallbloke), GO FOR IT but please make sure reviewers are NOT anonyms plus once the paper is published ALL review comments should be available to the public.
It would be a two edged sword.
Interesting comments at the Nature website following Dr. Ravetz’s article. I don’t see how Ravetz could have paid WUWT a higher compliment than he did in defending the use of blogsites in scientific debate. He names WUWT and Judith Curry in the following response to two comments that were as highly critical of his ideas as we were when he posted here several times. One of the critics argued (Climategate debates as an example) that blogsites ‘will not be ‘civil’, they will not debate in good faith, they will continue to misrepresent and poison wells in the guise of ‘sceptics’ or ‘critics’ just as long as it gets them the attention they need.”
Jerome Ravetz said:
There are important climate-science blogsites of a generally critical or questioning orientation where the courtesy rule is respected. Judith Curry initiated the courtesy rule in the course of a debate over the significance of Hurricane Katrina for global warming, and it is maintained on her website, see http://judithcurry.com. Anthony Watts’ site http://wattsupwiththat.com has vigorous discussions but again well within the limits of courtesy and mutual respect. That may be one of the reasons that it has been named ‘Best Science Blog’ last year and previously. Although talking (and listening) to bad people is well recognised as essential for the resolution of power-political disputes (see Northern Ireland and South Africa), it is, I recognise, a very new and strange idea for science-political disputes.
@bwanajohn 1:00 pm:
“I am in quality assurance and I can tell you it has nothing to do with trust and everything to do with verification. When I get a shipment of parts in, the first thing I do is open the box and verify it is what was ordered – that’s qualtiy assurance. You get into trouble when you ‘trust’ and not check at least once in a while. ”
LOL – We’ve all been bitten when we have NOT done that, haven’t we?
I know someone who bought several tens of thousands of pairs of shoes in China – only to receive a shipment of rocks, and the production plant miraculously disappearing into the night.
As Larry D. stated : True peer review is openly presenting the data of an experiment and methodology, then having another scientist attempt to duplicate the results. If the results of the first “scientist” can not be duplicated then a careful analysis of why there are differences is required- it could be errors in both, it could be errors in only one, it could be slight differences in procedure, or it could be errors in the basic hypotheses.
As we have learned by the Hypotheses of “greenhouse gas effect” there is Hugh errors in the hypotheses. The Hypotheses has never been put to the test of real experimental examination until my experiment which is in the process of being reviewed.
Anyone wanting to see an early draft of the experiment see http://www.The Great Climate Clash.com archives December, 2010, G3 “the Greenhouse gas effect does not exist” section 10 the demonstration
I am waiting for a few hundred physics students to attempt this simple experiment and reporting there results.
Words don’t mean anything experimental result speak volumes!.
@J Calvert N 1:23 pm:
>blockquote>I smell a Ravetz!
From wikipedia : “Post-Normal Science is a concept developed by Silvio Funtowicz and Jerome Ravetz”
(Trying to keep this on topic. Bear with me…)
I thought this was a joke, but looked up “Post-normal Science” on Wikipedia and there it was. It had two pertinent points, one which argues against Dr. Ravetz’s
From Wikipedia:
So, Dr Ravetz’ Post-normal science does NOT rely on trust, but puts in place as much dialog within “an extended peer community” as can be brought to bear. And the reason? To assure “the quality of the process and of the product.”
This is exactly what Anthony’s Web 2.0 approach has as its goal.
The second point about post-normal science in Wiki:
The last sentence requires a proper delineation between normal and post-normal science. And who is to be the judge of that demarcation?
I can see the reason for Dr Ravetz’s post-normal science approach, but that term “extended peer community” needs to be as “extended” as possible in the earth sciences when they are used “in service of political goals.”
So, Ravetz’s quote by Anthony is a good one
And Dr Curry’s
Hear, Hear! On the extended peer community.
I know from reading that The Royal Society in its early days was a raucus affair. Hooke and Newton hated each other, so you can guess what went on there.
But also Louis Agassiz’ early podium presentations about Ice Ages were pretty much ripped to shreds.
Raucus? Science is supposed to be raucus. Hooke could not stand it when people would present ideas without anything empirical to back them up.
Science is basically a school-yard scene and “Put up or shut up” should be the name of the game. How can it be SCIENCE otherwise? You put up, and you shut THEM up. Even if you DO trust someone, all that does is give them your ear. Even after they get your attention, they still have to get past your skepticism.
The existing Journals have acted as our “trusters of choice” for long enough. Now that they’ve lost the trust of a lot of people – with their version of post-normal science – how broke the system is has hit us smack in the face. Anthony’s Web 2.0 remedy seems like a doable approach to going forward from here.
Will it work? Will it fail? What does that matter at this point? It is an idea whose time has come.
When “trust me” is uttered the science has ceased and something else has taken its place. This is not a bad thing – in fact it is necessary since the science can rarely speak for itself. This website has earned its trustworthy reputation and I look forward to seeing some scholarly work pass through its gauntlet.