I’ve been made aware of two different opinions on state of science publishing as it relates to peer review and the pressure to publish even faster due to the Internet and all of its “instalaunch” tools.
First, in Nature, a comment by Dr. Jerome Ravetz: Sociology of science: Keep standards high.
He argues for embracing the new medium, while maintaining quality:
As more people become involved in online debates, quality need not fall by the wayside. It is encouraging to see that well-conducted discussions of points of contention between the scientific mainstream and critics are emerging, as the Berkeley Earth Surface Temperature study demonstrates (see Nature 478, 428; 2011).
Ultimately, effective quality assurance depends on trust. And science relies on trust more than most institutions. As Steven Shapin, a historian of science at Harvard University in Cambridge, Massachusetts, showed in his 1994 book A Social History of Truth, trust is achieved and maintained only by mutual respect and civility of discourse. In a digital age, civility should be extended to, and reciprocated by, the extended peer community.
Scientists have a special responsibility, but also a special difficulty. When their training has been restricted to puzzles with just one right answer, scientists may find it hard to comprehend honest error, and may condemn those who persist in apparently wrong beliefs. But amid all the uncertainties of science in the digital age, if quality assurance is to be effective, this lesson of civility will need to be learned by us all.
Dr. Judith Curry has some thoughts on this here, she writes:
I am a fan of the concept of “extended peer community” put forth by Funtowicz and Ravetz. Also, Ravetz’s phrase “the radical implications of the blogosphere” has definitely stuck in my head. Re the civility issue, I agree some level of civility is needed. Some think that Climate Etc. is too raucous (a not infrequent complaint made at collide-a-scape). A fair place for an honest debate might not be especially courteous. But the blogosphere enables a range of different types of fora and moderation rules. The challenge is to extract signal from the noise. I am pleased that sociologists are studying this.
At the same time, we have an editorial in Nature Geoscience, Embargoes on the web stating that scientists are increasingly acting as reporters now, and as a result, sometimes run afoul of publication rules. I see this as a shot across the bow against such practice.
Now that researchers, too, are acting as reporters, the guideline for talking freely to scientists but not to journalists may sound contradictory. Who should count as a member of the media for the purpose of the Nature journals’ embargo policy? The same basic rule applies: if an author actively seeks media attention before publication, we consider this a breach of our embargo policy.
At the same time, it is important to Nature Geoscience and fellow Nature journals that the scientific debate does not stop while a paper is under consideration. This principle also remains: we want our authors to present and discuss their results at conferences and communicate them to their peers. So if someone in the audience — journalist or scientist — tweets or blogs about a talk, we will not consider it to be a breach of our pre-publication embargo (see also Nature 457, 1058; 2009).
Where they say:
…if an author actively seeks media attention before publication, we consider this a breach of our embargo policy.
This squarely applies to the pre-publication publicity stunts pulled by Dr. Richard Muller and his BEST team.
People wonder why I dropped my support for him (like the feckless Dr. Peter Glieck and his science B.S. of the year awards), the answer lies within the shenanigans he pulled after earning my trust to use my data. I had always expected my data to appear in a full peer reviewed publication, instead, Muller spewed it in Congress and in his own media blitz in releasing papers that hadn’t even run the peer review gauntlet.
It may take some time (and additional train wrecks like BEST) before scientists learn that they can be their own worst enemy with these sort of behaviors.
OTOH, I’ve been considering a web 2.0 peer review experiment of my own. WUWT now has the ability to offer a peer review service for articles and papers. It is a new feature I can activate into WordPress, and would allow comments by invited reviewers to be posted for authors prior to publication, so that articles can be evaluated by a broad base of techical readers prior to publication.
I welcome readers thoughts on this idea. – Anthony
Anthony: If I understand your Web 2.0 peer-review model, it would allow a network of scientifically savvy readers (affiliated by at least their readership of WUWT) to read the draft paper and critique it (anonymously or not) before it’s published. (Of course, in the new online world, “publication” is a bit of a continuum, less of a distinct event. One doesn’t see bundles of newsprint with still-wet ink being dropped off at kiosks all over town at 2 AM).
I like it. Both because you have a terrific model here for intelligent and constructive discourse, and you should continue to build it, and this is a logical extension of what you’re (we’re?) already doing. And because the world really, really needs leadership on the process, to get us away from the corrupt and broken oligopolies that trace their power back to the physical print model. The same problem is present in medical journal publishing, and I think the same kinds of solutions are emerging. Go for it.
Anthony, did Guttenberg ask permission of the Church before printing the Bible? I don’t know, he might have, but would that have stopped him? He was in it for the money.
You web 2.0 idea will be done. By someone. Do you want to be part of that “someone.” Go for it.
It is time to take this necessary step. Professional journals have had a lucrative stranglehold on the peer review process. That in and of itself may not be so objectionable. But they have been co-opted, dominated and intimidated by a relativelt small clique of Mann/Jones and their followers. This clique has used its undue influence to turn peer review into pal review. We see the results every day, where authors insert the obligatory reference to climate change, global warming, etc., no matter how inappropriate or far-fetched the reference may be.
Providing an alternate venue that is open to true peer review by numerous experts in the relevant field is good for all concerned. Only the journals will hate it. But then, they brought it upon themselves.
You would be adding to science. So many of your readers are scientists. This would be an excellent forum for reviewing papers. Peer review has lost some of its authority and cache with the heinous behavior of people like Mann, Hansen, Jones, Gore (although, I list him with reservations as he is not a trained scientist), et al, I do not trust what I read from so many publications now due to their fraudulent behavior.
Web 2.0 peer review…..
Quis custodiet ipsos custodes?
AleaJactaEst,
Well, we know who watches the watchers now, don’t we? And we know pal review is corrupt; evidence is found throughout the Climategate emails. At least with Web 2.0, invited experts would be free to hash out any problems no matter what their point of view, and without interference from anonymous gatekeepers. Really, it’s all good.
Trust in peer review will only be gained if there is a high level of confidence in the standards of peer review and an acknowledged humility concerning the limits. The lack of trust is due to the highly partisan ways that peer review has been used to enforce consensus opinions, and the false/exaggerated claims made for peer review. In particular that:-
1) Peer review is the unambiguous demarcation between science and non-science
2) The peer-review process is sufficient to establish the thesis within a paper as the most advanced thoughts on a subject.
Lack of civility in discourse has the same routes as civil unrest within society. When a group in power has a tenuous grip on power, and excludes other groups by perceived heavy-handed and unfair means, then frictions will arise. Civility comes from recognizing other points of view, even those one feels firmly in error.
Peer reviewers are appointed by the publishing companies or by the scientific societies on the basis of their scientific credentials. They are under time pressure, don’t earn nothing in doing it, but the pride to be a reviewer. Some cronyism may take place but this is just human.
Peer review is a kind of quality assurance scheme:
a) Reviewers are asked to ascertain that the methodology presented in a paper is not absurd, doesn’t contain gross errors, to request more details when the description of the work looks fuzzy or incomplete, and to verify that the conclusions are plausible. They can recommend to refuse a bad paper. But let’s not assume that a peered review article is correct in everything because it has been reviewed.
b) It induces some quality concerns to the authors who know that peers will review their work, thus ensuring a kind of auto-censorship before publication (like knowing that radar controls on the highway will induce us to respect the speed limits).
I don’t see how a “web 2.0” review would replace or enhance the existing and needed peer review. How would be reviewers appointed? on the basis of which credentials? by whom?Would they do the same quality control as the current reviewers? Or would they engage in an early judgmental exercise without contribution to scientific quality but to an opinion debate. Would it result in an anti-peer review? Or just the same as a blog discussion?
I think it’s a great idea. The alternative news media has already had a big negative impact on monopolised global news outlets, particularly newspapers. Do the same with the published scientific journals, that’s what I say – make them irrelevant and, goodness knows, we’re well aware of how riddled by corruption they are after CG 1 & 2.
So WUWT has the credibility and wide global circulation (as evidenced by recent stats) to be the perfect forum for said reviewers.
This is going right up against the establishment though. They won’t like that. Might get SOPA on your back. All those clips from commenters will have to go, etc.. And look what happened to TB…
Hi Anthony
It seems a logical step forward despite the anticipated outcry from the existing Journals. Do you have thoughts about how reviewers would be nominated and chosen?
Anthony,
I know there were some wrinkles, but I see nothing wrong with Muller wanting the paper to have both regular peer review and open peer review. That way, they can address any major objections people come up with.
Whatever difficulties or dilemmas stand in the way, resources will be found to overcome them. This is an idea whose time has come.
Go for it!
Smokey,
Can we tar all of science with the same brush and do we go further and apply Web 2.0 to all scientific peer review?
What state is peer review in the hallowed halls of physics, chemistry, natural sciences etc.. & can we apply the precautionary principle that our detractors in CAGW are so beloved of? Be careful what you wish for.
On the other hand, it is said that human development moves in steps that are either evolution or revolution, the latter coming out of left field. Perhaps we ourselves inhabit left field?
peer – a person who is equal to another in abilities, qualifications, age, background, and social status.
….status quo
I’ve never understood why it would work in the first place….
……….I would want someone smarter than me to review my work
An open review (which can’t be done with even current peer review) with more ‘peers’, in an invite only setup, seems superior then the current system.
Even taking the ‘web’ out of it, the more reviewers the better, a more aggressive review system with more transparent comments, and various outlying experts involved would be better what any magazine or journal could do with 3 reviewers.
The idea as proposed could only be superior. There will be some things to iron out, but the wider the reviewer base, the wider the expertise, the better then end result. With the Internet, we could produce some very good science, the likes of that never has been as well reviewed.
Please, please, attempt it. Science could use the ‘Crowd source’ approach instead of the 3 experts in a dark room, who’s bias might not be apparent. Even if the first attempt fails, what will be learned from the process would make the next attempt even better.
Best of luck!
One of the reasons for lack of civility in scientific discourse is the deliberate exclusion of dissenting views, and making claims that are not sufficiently backed by data and method to enable replication. As an example, please see Steve McIntyre’s posting of yesterday on the partisan withholding of data by Prof Phil Jones.
http://climateaudit.org/2012/01/08/nature-and-the-inundation-legend/
In particular this quote from one of Phil Jones’s emails
“Years ago I did send much paleo data to McIntyre but have also had nothing but criticism on his blog ever since. As I said, this criticism on blog sites is not the way to do science. If they want to engage, they have to converse in civil tones, and if people don’t want to work with them, they have to respect that and live with it.”
In other words, to engage one has to refrain from criticism, or one should accept exclusion. Much like a Gentlemen’s club of a bygone era, but without the standards.
Since there’s a general tendency for warmist expertets to avoid this site, I think your idea for “peer review” would be just as much “pal review” as the current “Team” approach. I would be skeptical about whether the results would be of any higher quality than current methods.
From my researches in super-conductivity I know academics talk a lot of rubbish.
Open review of current scientific theories will almost certainly trigger advances across all disciplines.
And frankly this site is the perfect initial host (until it gets too big 😉 ).
Latitude says: January 9, 2012 at 11:03 am
And suppose there IS no one smarter?
Anthony,
Lean forward…
No new system will work unless there’s money pulling in the proper direction.
Quality control currently works well in areas where good science leads to salable products, or where good science leads to fewer lawsuits. Materials science, agricultural research, engineering, medicine.
Quality control fails in areas where bad science leads to bigger grants. Climatology, quantum physics, “social sciences”, cosmology.
And where money or lives are not involved, quality control frankly doesn’t matter. It’s just fiddling around anyway, so quality is moot. If it’s fascinating to somebody, it’s OK.
Latitude says:
January 9, 2012 at 11:03 am
……….I would want someone smarter than me to review my work
——————————————————————–
Well said. Most people here are smarter than me: that’s why I come here.
(Mods, I left a comment about this on the milestone thread but I think it may have been spam-binned – don’t mean to make a fuss but it’s just that it is relevant to this thread. Alternatively, I’m going mad. Cheers).
[REPLY: It’s not there. Resubmit, if you wish. -REP]
Do it.
The internet is far more open and so less corruptible and probably immune to the setting in of group-think.
Some publications have been found wanting in recent years and some have succumbed to lower standards. Whilst some (Nature ?) have taken worthwhile steps to improve their readers confidence in their standards, I feel the that the future lies with the people and the internet.
I am not a scientist and will probably never present a paper. But if I were to present a paper to the World, I would never dream of going to a print magazine. For me, publishing my paper on WUWT or Tallbloke’s Talkshop, would be far more satisfactory, allowing me to gain a wider readership and to interact with those readers and answer questions.
We see this sort of thing more and more, with major scientific figures presenting initial papers on WUWT and Tallbloke. The internet is without doubt the future of scientific publishing and peer review, prior to subsequent publishing on the main site.
Do it.
From the Ecological Society of America: Fighting back against open access.
A telling quote:
One way to make taxpayer funded research more visible and accessible to interested members of the public would be to require federally-funded grantees to provide a second version of the research summaries they already prepare, specifically for the lay reader.
I guess that we’re not smart enough to read the real thing. /sarcasm
http://www.esa.org/pao/policyStatements/Letters/ESAResponsetoPublicAccessRFI2011.pdf
I welcome readers thoughts on this idea. – Anthony
=====================================
Why shouldn’t peer review evolve like everything else?
The system that is in place right now, is simply because of the paper work (literally) involved.
We did away with the paper work years ago………
I’m not really comfortable with peer review by like minded souls. Its my competitors and rivals who are my harshest critics.
If there is a weakness or a fault, they’re right on it, no holds barred.
If you want supportive views, ask your Wife, Friends, Co-researchers.
If you want a scales-off perspective, ask your Mother-in-law or Teenage child.
“We did away with the paper work years ago………”
ok, well one other thing…..
kept open and honest…..the internet can rip a new butt hole in most of the garbage that’s being rubber stamped today……….
A journey of a thousand miles begins with a single step.
An excellent idea whose time has come. Public discusssion has the potential to greatly further the search for answers. In any case, the old paradigm is in failing health … keeping it on life support will help no one.
I love the idea. Though quite frankly, if I write a paper, I don’t care if the person that stops me from stepping on myself in public has a PhD or not. Someone who just finished undergraduate statistics might actually be the one to find that bias that crept into my analysis. I’d rather be right than dignified in the end.
Leon Brozyna says:
January 9, 2012 at 12:05 pm
I read that originally as “A Journal of…” they say reading is fundamental, but perhaps that is what we are seeing: the birth of a new Journal online.
My suggestion is to have a broad range of reviewers since “climate science” touches on so many disciplines. Research that uses tree-rings ought to be looked at by botanists, for example, just to catch unfounded assumptions by non-biologists and identify weaknesses in an hypothesis. The group obviously will include technically competent reviewers, but shouldn’t be limited to them. Communications specialists as well as mathematicians are needed.
I think I am missing something, like how it will work.
1. Does each reviewer get one vote? (If so, why is it not a popularity contest?)
2. How long does the review process last?
3. What will the review process entail?
4. With each specific criticism, what constitutes adequate correction? Agreement by just that reviewer and the author?
5. What is the method of formal criticism?
6. The whole process could easily become unmanageable discussion or argument.
7. Who decides what the final form shall be?
8. Will editors/referees/moderators still be necessary? If so, how chosen?
9. Will any minimum standards be applied?
10. I assume the structure will be posted on a link.
11. Will accepted papers be listed with such central clearing houses as Springer or Wiley or Mendelay?
12. Will papers be behind a paywall? (Nothing is stupider than 80-year-old papers that some journal expects one to pay $30 for.)
13. How is at all paid for? Reviewers pro bono still? Editors? Ads? (Ads wouldn’t be a bad idea.)
14. Pre-publication media hype – what effect would it bring down upon the head of the researcher?
15. The biggest one maybe: Will secrecy be done away with? (In all the years of the existing peer-review process, what was really behind keeping the reviewers’ identities secret? A: Shouldn’t they have been given credit? B: Didn’t the researcher have the right to face his accusers?) With open-peer-review (OPR), all the cards are on the table. It seems like a much better system.
Certainly that list does not exhaust the pertinent points to address.
All that said, there is no one forcing any researcher to submit to any one journal. This means more or less that the journals that currently exist more or less just declared themselves to be a journal, then sat back and waited to see if anyone would submit papers.
Bottom line? There is no harm in trying.
I agree – Go for it!
I am wondering if anyone can assist me. I am trying to find historical weather records for Winnipeg and Regina but Environment Canada seems to have lost the data. It happened about 2 years ago, when they said at the time that they were revising the data.
@Stephen Rasey
I hope you mean Gutenberg. (Zu) Guttenberg is a synonym for blatant and unrepentant scientific plagiarism in Germany.
@Gary Young 12:23 pm:
“My suggestion is to have a broad range of reviewers since “climate science” touches on so many disciplines. Research that uses tree-rings ought to be looked at by botanists, for example, just to catch unfounded assumptions by non-biologists and identify weaknesses in an hypothesis.”
Why not open it up to whomever has any input at all?
In the beginning of The Royal Society, it was a meeting in a real room/hall, and the members would bring their experiments and run them, right in front of everybody – biologists, geologists, astronomers, physicists, etc. – and everyone had a free shot at the presenter.
Isn’t Web2.0 (which includes blogging, but would also include this set up) essentially the same thing as an open forum? “Forum” means both the PLACE of a meeting or interaction on one hand, and the open interchange itself. “Forum” implies both a level of decorum AND a level of chaos. There has been too much decorum in Journals, so much that they have become ossified, too crystallized. It is time for a new form, an open forum, just as Anthony is considering.
Anthony, I would propose that the name include the word “Forum.” As in “The Science Forum” or some such.
As the BEST SCIENCE BLOG OF 2011, and a perennial candidate, WUWT has at least a tiny standing from which to build.
From the scientists point of view, it will not get much recognition unless you can convince someone to include it in their index. For topics related to living systems PubMed might work, but something else might be needed for purely physical/chemical studies.
Nothing new here, but I do want ot point out that Nature and Scientific American have been a major part of the problem with both of them not only promoting incivility and intolerance, but junk science.
Go for it, Anthony.
You built WUWT as a civic model providing open discourse about “climate science” among the interested — in the interest of mankind, which ultimately pays as science and policy seemingly are led astray together. WUWT has brought together scientists and interested non-scientists in a civil, well-informed discourse regarding whether the Emperor is wearing clothes, or not.
I trust that you will bring the same civility, transparency, and honest heart to the attempted upgrade of a clearly broken peer review process.
I am in quality assurance and I can tell you it has nothing to do with trust and everything to do with verification. When I get a shipment of parts in, the first thing I do is open the box and verify it is what was ordered – that’s qualtiy assurance. You get into trouble when you ‘trust’ and not check at least once in a while.
Peer review should work the same way. Data and methodologies should be provided so that the reviewers have enough information to verify what is claimed – plain and simple.
Anthony,
I think your web 2.0 project is a great idea. Go for it.
Anthony, “a Web Peer Review”. What a Great Idea! The old system is broken and was hijacked by those in it for the money.
You web 2.0 idea will be done by someone. You. yourself need to do it now!
From Dr Ravetz: “Ultimately, effective quality assurance depends on trust. And science relies on trust more than most institutions. ”
—————————————————–
This makes no sense to me. The scientific method relies on replication of results, not trust. Just as the scientific method includes skepticism of claims, not trust in the scientist making claims.
I smell a Ravetz!
From wikipedia : “Post-Normal Science is a concept developed by Silvio Funtowicz and Jerome Ravetz”
** Rant on **
If science was like trading in equities then maybe Ravetz would have a leg to stand on. But I see science as an epistemology, a way of knowing. The only thing that makes science internally valid as an epistemology is the ability to replicate the work! If you can’t replicate the work, then it might be “social science” (p < 0.05) but it isn't science.
** Rant off **
With my rant now over, I fully encourage you to go full steam ahead on the web-based peer review project. Based on all of the unofficial volunteer reviewing that is performed on your fine website it seems there already is a willing pool of reviewers ready to go at it.
There are a lot of mistaken impressions about peer-reviewed journals. The peer review is just an editorial quality check, but the reviewers don’t try to re-create the graphs, or check the math or the methodology, they just check the text. The real quality check is when other people try to replicate the results.
I agree with ‘Height of Nonsense’ (Tallbloke), GO FOR IT but please make sure reviewers are NOT anonyms plus once the paper is published ALL review comments should be available to the public.
It would be a two edged sword.
Interesting comments at the Nature website following Dr. Ravetz’s article. I don’t see how Ravetz could have paid WUWT a higher compliment than he did in defending the use of blogsites in scientific debate. He names WUWT and Judith Curry in the following response to two comments that were as highly critical of his ideas as we were when he posted here several times. One of the critics argued (Climategate debates as an example) that blogsites ‘will not be ‘civil’, they will not debate in good faith, they will continue to misrepresent and poison wells in the guise of ‘sceptics’ or ‘critics’ just as long as it gets them the attention they need.”
Jerome Ravetz said:
There are important climate-science blogsites of a generally critical or questioning orientation where the courtesy rule is respected. Judith Curry initiated the courtesy rule in the course of a debate over the significance of Hurricane Katrina for global warming, and it is maintained on her website, see http://judithcurry.com. Anthony Watts’ site http://wattsupwiththat.com has vigorous discussions but again well within the limits of courtesy and mutual respect. That may be one of the reasons that it has been named ‘Best Science Blog’ last year and previously. Although talking (and listening) to bad people is well recognised as essential for the resolution of power-political disputes (see Northern Ireland and South Africa), it is, I recognise, a very new and strange idea for science-political disputes.
@bwanajohn 1:00 pm:
“I am in quality assurance and I can tell you it has nothing to do with trust and everything to do with verification. When I get a shipment of parts in, the first thing I do is open the box and verify it is what was ordered – that’s qualtiy assurance. You get into trouble when you ‘trust’ and not check at least once in a while. ”
LOL – We’ve all been bitten when we have NOT done that, haven’t we?
I know someone who bought several tens of thousands of pairs of shoes in China – only to receive a shipment of rocks, and the production plant miraculously disappearing into the night.
As Larry D. stated : True peer review is openly presenting the data of an experiment and methodology, then having another scientist attempt to duplicate the results. If the results of the first “scientist” can not be duplicated then a careful analysis of why there are differences is required- it could be errors in both, it could be errors in only one, it could be slight differences in procedure, or it could be errors in the basic hypotheses.
As we have learned by the Hypotheses of “greenhouse gas effect” there is Hugh errors in the hypotheses. The Hypotheses has never been put to the test of real experimental examination until my experiment which is in the process of being reviewed.
Anyone wanting to see an early draft of the experiment see http://www.The Great Climate Clash.com archives December, 2010, G3 “the Greenhouse gas effect does not exist” section 10 the demonstration
I am waiting for a few hundred physics students to attempt this simple experiment and reporting there results.
Words don’t mean anything experimental result speak volumes!.
@J Calvert N 1:23 pm:
>blockquote>I smell a Ravetz!
From wikipedia : “Post-Normal Science is a concept developed by Silvio Funtowicz and Jerome Ravetz”
(Trying to keep this on topic. Bear with me…)
I thought this was a joke, but looked up “Post-normal Science” on Wikipedia and there it was. It had two pertinent points, one which argues against Dr. Ravetz’s
From Wikipedia:
So, Dr Ravetz’ Post-normal science does NOT rely on trust, but puts in place as much dialog within “an extended peer community” as can be brought to bear. And the reason? To assure “the quality of the process and of the product.”
This is exactly what Anthony’s Web 2.0 approach has as its goal.
The second point about post-normal science in Wiki:
The last sentence requires a proper delineation between normal and post-normal science. And who is to be the judge of that demarcation?
I can see the reason for Dr Ravetz’s post-normal science approach, but that term “extended peer community” needs to be as “extended” as possible in the earth sciences when they are used “in service of political goals.”
So, Ravetz’s quote by Anthony is a good one
And Dr Curry’s
Hear, Hear! On the extended peer community.
I know from reading that The Royal Society in its early days was a raucus affair. Hooke and Newton hated each other, so you can guess what went on there.
But also Louis Agassiz’ early podium presentations about Ice Ages were pretty much ripped to shreds.
Raucus? Science is supposed to be raucus. Hooke could not stand it when people would present ideas without anything empirical to back them up.
Science is basically a school-yard scene and “Put up or shut up” should be the name of the game. How can it be SCIENCE otherwise? You put up, and you shut THEM up. Even if you DO trust someone, all that does is give them your ear. Even after they get your attention, they still have to get past your skepticism.
The existing Journals have acted as our “trusters of choice” for long enough. Now that they’ve lost the trust of a lot of people – with their version of post-normal science – how broke the system is has hit us smack in the face. Anthony’s Web 2.0 remedy seems like a doable approach to going forward from here.
Will it work? Will it fail? What does that matter at this point? It is an idea whose time has come.
When “trust me” is uttered the science has ceased and something else has taken its place. This is not a bad thing – in fact it is necessary since the science can rarely speak for itself. This website has earned its trustworthy reputation and I look forward to seeing some scholarly work pass through its gauntlet.
It would be good for science if all comments (as well as the data) were made with respect to the peer review of papers were public. Moderated blogs have an opportunity to do this. If understood correctly, Web 2.0 seems to support this.
Inspired thinking Anthony.
As many have already said; an idea whose time has come.
The peer is dead. Long live the peer.
I think it is a great idea. Bring discussion out into the open. Isn’t there some old quote, “Secrecy is the mother of corruption,” or something like that? It is only when things are kept away from fresh air and sunshine that they start to stink.
I think open discussion already is happening on this site. Of course, you now have to put up with all sorts of cracks from the peanut gallery, here. However, if peanuts like myself are not allowed to speak at a new site, may we please still lurk?
Perhaps the biggest worry would be the timid sort of scientist who doesn’t like all the slamming and banging of frank exchanges of ideas. As I recall, Darwin withheld his ideas for quite a while due to such timidness. Was it Huxley who was Darwin’s advocate? It always helps to have a brawler on your side.
As a person who has brought forth a number of incorrect ideas on this site, I didn’t mind learning where I was mistaken. “It is better to stand corrected than to fall for flattery.” In fact the WUWT peer-review seemed downright civil, especially compared to other sites. However even here it can be a bit rough, especially if one is gentle-hearted. Speaking for myself, it took me years to thicken my skin, (and I still lose it on occation.) As a teenager, at the slightest sign of critisism, and I was long gone. I imagine there are some scientists who simply wouldn’t want the fuss, and who’d rather toil in obscurity. You’d have to drag them out.
Another benefit of on-line peer review such as Anthony proposes would be the limiting of the Chump Effect, where credulous reporters uncritically accept whatever mainstream climatologists tell them regarding the latest looming disaster.
It used to be that journos would print both sides of the story. But as they say, that was then and this is now. Now we need a Web 2.0 to sift the wheat of truth from the chaff of grant-fueled globaloney propaganda. Because the media no longer does its job, and climate journals have been corrupted. Time for an end run, since the Establishment will not police itself.
Sounds like a great idea until you realize that few if any of the Climate Science establishment will have anything to do with a WUWT journal either as contributors or reviewers.
Also, open review, if that is what you have in mind, will prove largely pointless because those who volunteer comments are generally not peers in the sense that they have substantial research experience in the field to which the submitted article relates.
Several factors account for the weakness of peer review:
First, in many areas of research, there really are few peers and they may not be that bright. In that case, reviews aren’t that bright either.
Second, top scientists will often decline to review stuff by novices, so poor work often gets an fairly easy pass from reviewers as inexperienced or clueless as the author(s).
Third, in some fields, cronyism develops, where colleagues and co-workers manage to edit or review one another’s stuff.
Fourth, many scientists accept editorships mainly because it beefs up their CV, not because they intend to spend weekends and evenings reading every manuscript that crosses their desk and delving into the literature to find qualified reviewers and to enable them to make their own independent judgment of the work in hand. As a result, editors often allow or encourage cronyism, consulting only their friends and colleagues as reviewers, and then accepting the reviewers’ recommendation without question rather than doing the hard work of making their own assessment in the light of the reviewers’ comments.
A good journal needs a clever and broadly experienced editor with a commitment to science and a readiness to encourage and assist the inexperienced while, as necessary, standing up to pressure from the big names in the game. Such individuals exist and create the great journals. Anyone who’s unwilling to play that role will never edit a first rate science journal and might as well not try.
From the post;
OTOH, I’ve been considering a web 2.0 peer review experiment of my own. WUWT now has the ability to offer a peer review service for articles and papers.
There must be lots of questions.
Would it be a fully open review only process, like happens here now?
Or would it be a part closed access with “relevant” specialists encouraged to examine or replicate parts? This may be much quicker but potentially could suffer the biases and ultimately editorial mis-use that some journals have seen.
PLoS ONE is an entirely online journal offering quick review and publication that I look at often.
They are also asking questions about their own quality control and their effectiveness (and ranking?). Perhaps there are some other online examples already in existence to look at too?
The fully open process is slow and risks losing track of where it is up to.
For example I take Lord Moncktons reiteration of the “effective radiation temperature” (255K for Earth) as criticism of Nikolov and Zeller’s poster. Dr Brown debunked that by showing that the average temperature (of an atmosphere-less world with large surface temperature variations) must be much lower than the “effective radiation temperature”.
Willis Eschenbach examined that further by looking at some actual temperatures on the moon (compared to the moon “effective radiation temperature” of around 270 K) and came up with an average temperature of 196 K (in a rather sun exposed part).
So far it looks like Nikolov and Zeller integrating over the entire surface of the moon (including the colder polar regions) may be right with their average of 154 K.
If people disagree of course someone might like to replicate their result?
Willis started to look at adding atmosphere. His imagined atmosphere that is perfectly transparent to infrared will not allow the surface temperature to exceed the “effective radiation temperature”, suggesting he disagrees with Nikolov and Zeller (and Hans Jelbring as raised by Tallbloke).
I would suggest that no atmosphere is perfectly transparent to IR, and I seem to remember Nikolov and Zeller, and Hans Jelbring qualifying their atmospheres to be “thick enough” (so it intercepts a significant part of the IR going through). I better re-read them.
So, Anthony, if you can get more feedback on the atmosphere effects, then the first paper will have had its first stage review.
Or is the Nikolov and Zeller post of their poster too far back (dozens of posts and thousands of comments) for people to keep track of it.
Then how will the edited version be examined (and over how long?)
And how would it be finally “published”?
@John09
“And how would it be finally “published”?
That’s an excellent question.
Even when a paper passes peer review there will remain, in the great majority of cases, many mistakes. Thus even after acceptance in principal, a good editor will mark-up a paper pointing out errors and suggesting, as appropriate, improvements to the text, the figures, the tables.
Then, when the “final draft” is submitted for publication, a conscientious publisher will have a skilled copy editor give the thing a further work over. This may deal only with spelling, and punctuation, formatting of references, etc., or it may involve checking that what is stated in the text matches what is shown if figures and tables, checking the math, making sure that what is said here corresponds with what is said there, correcting or standardizing the format of equations, redesigning tables, and perhaps re-lettering or redrawing figures.
Today, most such work is off-shored to a low wage jurisdiction, but in that case, the work is likely done in a perfunctory way, dependent primarily on the use of software, such as grammar and spelling checkers, i.e., without input of judgement or editorial expertise. This is how the oligopolistic journal publishers now mainly do the job. There remain, however, some publishers, including the AAAS, publisher of Science Magazine, that still employ smart, conscientious in-house editors who add substantial value to what is published. But such editing in North America, cost real money.
CanSpeccy,
You seem to be forgetting the primary reason for peer review: to falsify conjectures and hypotheses. If new ideas cannot be falsified despite the best efforts of scientific peers, then they are on their way to becoming accepted theories. But science has taken a back seat to a new agenda: diverting as much public money as possible into promoting the AGW narrative.
The ossified journal review process has finally run out of steam, because it has become entirely self-serving. Scientific journals are big business, and climate journals more so, given the vast amounts of public tax money paid out annually to “study climate change”. You are right that the journal business is an oligopoly. But it is ripe for breaking:
“You can resist an invading army; you cannot resist an idea whose time has come.”
~Victor Hugo
B.Klien: I can’t get your link to work. Could you please see if it’s ok? Thanks
“You seem to be forgetting the primary reason for peer review: to falsify conjectures and hypotheses.
Not so.
The sole purpose of peer review is to determine whether the evidence presented and the analysis based upon it are valid. What that means is, assessing whether the methods are theoretically valid, the results theoretically consistent with the methods and the conclusion logically drawn.
The business of falsifying hypotheses, assuming that were the basis of scientific progress, a proposition that may be doubted, is the job of the people who write the articles that are published in peer-reviewed journals.
For example, a paper that refuted the hypothesis of AGW would be assessed by the reviewers for validity of methods and logical inference. But it would certainly not be the reviewers’ task to refute the work by making independent observations or undertaking a novel analysis, for if they were to do that, their work would constitute, in itself, a new contribution that would be deemed in need of peer review.
CanSpeccy – You make some good points about the reasons for the weakness of peer review.
The fact that “few if any of the Climate Science establishment will have anything to do with a WUWT journal either as contributors or reviewers.” and open review, if that is what you have in mind, will prove largely pointless because those who volunteer comments are generally not peers in the sense that they have substantial research experience in the field to which the submitted article relates.” I think will become of limited significance.
True Peer review has its place, but it is not the be all and end all. I think there is a place for wider review. This is because smart people are smart people regardless of their area of specialisation, and sometimes someone from the outside will quickly spot a glaring fault that those too close to a problem are blind to.
The other reason I feel inclined to disagree is because the argument for specialised peer review bear too stark a similarity to the argument used by the Catholic Church as to why the Bible should be in Latin and only the Clergy allowed to read and interpret it.
The apt response of William Tynale to that argument was
“I defie the Pope and all his lawes. If God spare my life, ere many yeares I wyl cause a boy that driveth the plough to know more of the Scripture, than he doust.”
Translation and the printing press killed that argument 500 years ago fro religion.
Personally I suspect that, given time, open review through the internet may well do the same for science.
In a way it has already started. The fact that scientific papers and reviews and discussions about them are already available on the internet means that the geni is well and truly out of the bottle and is not going back in.
dibs on name —–> WordPress SPeeR (scientific peer review)
Clever I know. Please hold the applause, I’m humble.
This can actually be a business model, at worst perhaps WordPress would be persuaded to get involved since the realm of science is large, the field focus would be required for efficiency. Categorizing, sub-categories, multiple threads within a Paper to streamline and localize discussions, minimal membership fees, application fee, non-invasive advertisments, etc. Sounds so beautiful, if only I wasn’t inept. Btw here’s another link from 11-20-2011 about this :
http://www.readwriteweb.com/archives/the_internet_will_get_a_peer_review_layer_next_yea.php
play find the faults in that article, it’s fun, and numerous.
“WUWT now has the ability to offer a peer review service for articles and papers. It is a new feature I can activate into WordPress, and would allow comments by invited reviewers to be posted for authors prior to publication, so that articles can be evaluated by a broad base of techical readers prior to publication.”
PLEASE! What would you call it? The Ultimate Acid Test? If ideas could survive review here they would be on great footing.
One point I do think you should consider. Double blind the setup. Neither the submitter nor the reviewer should know who each other is. I know most reviewers could figure out who’s paper just by looking at the evidence and seeing who was working on that but double blind is still a worthy goal.
The only way to effectively use what is in effect crowd sourcing on science is to limit the discussion to the science. Comments such as, “this paper uses the typical model used in this subject area”, or, “I recommend publishing” has no place in honest review of a paper. Identifying weaknesses, uncertainties, honesty in data representation and where possible replication of analysis, these are what help validate science.
I also believe that some journals would be willing to join in such an effort, at least as a trial. It would use this approach for the more controversial papers, providing a higher level of review prior to publication, but with the potential to move science further forward more quickly than traditional outlets.
There would still need to be an editorial board to select the papers deserving of such review and who must make the final decision on publication. Those would be journal duties and would improve the review process by not overwhelming the review community.
@Peter S
“I think there is a place for wider review”
Peter, I don’t dispute that for one moment. I think WUWT makes an important contribution by providing just that. Likewise, Steve McIntyre and others.
With luck, such review will compel science publishers to undertake their task with a greater sense of responsibility.
While a journal by a noted skeptic might attract some articles, it would be most unlikely to attract contributions from the mainstream climate science community, or be treated by them as anything much more than a propaganda effort.
I just don’t see how one could get around that problem, however great the integrity, open-mindedness and competence of the publisher.
True some people might contribute, and they might contribute interesting and worthwhile papers, but they would be branded as “the usual suspects” and it is unlikely that their work would attract funding or much mainstream climate science discussion.
It seems to me that it is best for those who perceive a warmist bias in the work of mainstream climate scientist to keep up the largely effective informal critical review.
CanSpeccy,
You’re right about the purpose of peer reviewed journals. They open the door to falsification, but their purpose is not falsification per se. But they are the self-appointed gatekeepers, and they have been compromised.
A.W. Montford made it crystal clear in The Hockey Stick Illusion that the climate journal/pal review system is heavily rigged against skeptical scientists. It is time for everyone to lay their cards on the table, and let the best hand win. Gaming the peer review system has had its day, and it is time to try something new. There will be problems, of course. But the current system is such a dishonest mess that the new imperative is to trash it and try something uncorrupted.
Anthony, an elegant solution, but possibly too simple to be valued!
I believe the correct term for this is “crowd sourcing”. Please check out
http://www.openideo.com/
Re: Journals as gatekeepers and the need for something new.
There have always been charges of reviewer bias, breach of confidentiality, and plagiarism, e.g., Peer review: the Holy Office of modern science.
So there is nothing unprecedented in the criticism that has been leveled at the review process of some Climate Science journals, although a consistency of the bias across the majority of journals maybe unprecedented.
So, in principle, I would applaud anyone rash enough to launch a journal with a declared lack of warmist bias. But on the question of starting a new journal, I admit to a prejudice. I managed peer reviewed journals for many years. It was interesting and sometimes rewarding, but it was certainly not the most relaxing vocation. So if anyone asked, my advice on launching a journal would be the same as that given by H.L. Mencken to a young man who asked advice on starting a literary magazine: “Young man,” he said “buy a revolver and shoot yourself.”
Go for it. Don’t see exactly how it gets paid for; that is one of several critical issues.
Points to consider:
1. The original has to have enough information – including new analysis of existing information – for reviewers to look at the original. Think of the slush pile at any magazine.
2. Submission must contain data, sources of data, error bars, etc. Words without math are rarely science. Central clearing house can recommend expertise for papers to get closer to completion. For climate/weather papers, this would always include professional-level statistics.
3. Need enough reviewers to be able to review any important aspects of the submission. Reviewers can work the slush pile and indicate yes/no on desire to review, with “no” given a quick reason – e.g. too busy, this paper is junk, too close to existing publications – not novel, my expertise is not in field covered by submission, math is over my head, data provided insufficient, math is over submitter’s head, obvious errors invalidate submission, no falsifiable predictions, etc.
4.. Once a submission gets a “will review” green light from enough reviewers, move discussion of pre-print to public area of web site – very much like WUWT of today. Comments should include factors omitted in paper, alternate statistical analysis to raw data, references to previous literature. Comments should minimize “me-too”, ad hominem, irrelevant/off-topic, with the option to see all. Idea is to let readers help keep signal-to-noise level high, without becoming an echo chamber.
5. At some point, need to finalize paper, with caveats and comments/discussion. That way, subsequent papers can refer to paper and comments while knowing that the previous publication is accessible and immutable.
6. Expect this to evolve into collaborative papers, as most of us don’t have all the expertise needed to do research, write using clear language, find flaws conceptual and mathematical, etc.
If done right, this could hit critical mass. If done poorly, could become theological, with different camps not communicating meaningfully.
Off topic: No poetry. I was an editor, once, and made the novice editor’s mistake of mentioning somewhere that my publication accepted poetry. Sturgeon’s Law ensued, with 90% being wildly optimistic. My revision to Heinlein’s Law (often corrupted to Hanlon’s Law). “A sufficiently advanced stupidity is indistinguishable from malice.” The converse is also true.
Thomas L.,
Thanks for the positive suggestions. Nothing will be accomplished that is easy. But it is time to sweep away the old, and embrace the new paradigm: on-line peer review.
This so much on the lines of what Anastassia Makarieva asked for in her plea for reviews here at WUWT. I’m not so sure this should not be a two step process.
One is to help through open discussions on a paper to help the authors to iron out the kinks that would surely mean a rejection if submitted for final publication without it. I have thought a couple of times in the past of publishing a paper but I would need such wide-based discussion, criticisms and suggestions. That aspect is not found anywhere today without picking a few colleagues which immediately jumps into the realm of group-think.
In such an arena, no arguments. Sorry Joel. Pointed comments and requests for clarification only. Quips, cuts and off-topics banned. But such process would have to be open to *anyone* who wants to inject pertinent thoughts, criticisms or suggestions (as my lengthy discussion with Dr. Makarieva on sailplanes and Oklahoma storms cloud action, she seemed to get valuable input from both). See, sometimes an author might get an idea to better their paper from “left field” as another commenter here just put it. If you have a process to pick ‘those in’ and ‘those out’ you automatically limit the good of such a procedure.
Two would be the formal peer review process whether status quo or on the web with limited qualified reviewers. I would so much like the cronyism to fall by the wayside. Maybe it should be a vote system of all actual scientists in the field picking the reviewers instead of the publisher, or maybe a round-robin within very specific specialties. That may not completely do away with the consensus aspect but it for sure would add some randomness and take the control and power away from a few powerful people.
We should be careful of exactly what level this process would address.
500 years ago the movable type printing press broke the control of the clergy on the dissemination of knowledge to the people of the world. But still those who would control knowledge only needed to control the press to control the dissemination of knowledge.
Today we have “the NET that covers the world”. At present anyone that wishes can communicate with any other that wishes to listen. Knowledge is now available to all and all can contribute.
Anthony has the bases and tools to create a new product Web 2.0 as a science publisher with real standing and wide spread review. I for one hope he moves forward with this concept. pg
Gary Swift says:
January 9, 2012 at 11:08 am
I would be skeptical about whether the results would be of any higher quality than current methods.
I have to agree. So long as peer review is in any way concerned with the conclusions of a paper it is bogus science. The one and only purpose of peer review should be to ensure the methods used in the study were sound.
However, that is not what we see in modern peer review. Instead we see “belief” at work, where the merits of a paper are judged by it conclusions, not its methods.
WUWT could open up a new standard in Peer Review 2.0 by allowing any and all review that speaks to the methods of the paper, and let the conclusions speak for themselves.
When relativity was introduced, many scientists of the day rejected the conclusions because they were at odds with their training and common sense view of the universe. However, they were not able to argue against the methods and over time repeated observation has shown relativity to be the most successful description of gravity available.
Imagine that in the early 1900’s a jealous scientists with a competing idea had prevented the publication of relativity, because it undermined their own work. Such an event is not uncommon in the history of science. Imagine that as a result the years of work that were required to complete the theory were never invested. Would schools still teach that gravity was a result of the flow of ether?
The debate over AGW and CO2 is no different. So long as any paper can be censored because of its conclusions, science will be held back. Peer review should only talk to the methods of a paper. Experiment and observation by someone actively seeking to overturn the theory is the true test of science. The very thing that Phil Jones did not want to see happen to his work.
CanSpeccy says:
January 9, 2012 at 6:48 pm
“You seem to be forgetting the primary reason for peer review: to falsify conjectures and hypotheses.
Peer review cannot do anything of the sort. That is the great misconceptions about Peer Review. That if a theory has been Peer Reviewed , the theory must somehow be correct. The problem starts with the scientists themselves that hold up peer review as though it meant the process in some fashion proves their pet theories correct. It doesn’t.
Interestingly, Skeptical Science is reviewing their Comments Policy (see: http://www.skepticalscience.com/2012-SkS-Weekly-Digest_1.html).
One wonders whether they might consider toning down after the inflamed rhetoric of their B S awards.
I suspect not from the general levels of satisfaction expressed by most contributors. The success or failure of any endeavours at wider peer review depend greatly on commitment to courtesy and consistency.
Judith Curry to her credit manages a very civilised forum with very light moderation whilst accommodating quite disparate views. While clearly accepting the premises of AGW, she seems equally willing to acknowledge the multiple uncertainties in the science. Her latest thread, Too Big to Know (see http://judithcurry.com/2012/01/09/too-big-to-know/#more-6528), courageously tackles the complexities too many of us seek to avoid.
WUWT to its credit selects many articles presenting scientific work from a pro, anti, and neutral perspective vis-a-vis AGW. Unfortunately, many comments take an overly curmudgeonly contrarian outlook (“Not more models!” would be a very mild example) rather than a considered critique. It’s role as a venting space for folks who are p*d off with the monolithic AGW “consensus” can undermine its credibility. Judith by contrast seems to manage to engage folks in the nitty gritty of the scientific debate and its philosophical underpinnings whilst tolerating an eccentric visitor or two.
Why not look at a shared endeavour with someone like Judith? I suspect the two sites could learn a great deal from one another.
Dan In California has understood the problem. Ravetz’s conclusions are fine and dandy – about time – but for all the wrong sociological claptrap reasons. Science is not ‘based on trust’. That would be….err.. Theology.
Now here’s the thing. Science uses the ‘replication’ to which DIC refers, to test its hypotheses, sometimes with startling results. Planck, for example, simply could not believe what he had previously thought to be truth ‘by law established’; and this disbelief forced him to change his own hypotheses. What a magnificent contrast he makes with so many modern ‘scientists’! And many of the most revolutionary ideas in Physics (in particular) came from nothing more than intuition, dreams, superstition or ‘metaphysics’. Einstein’s and Jeans’ and others’ ideas all share a Kantian climate whereby time, space and causality are no longer assumptions, but the subjects of an examination.
So truth is to be found ‘wher’ere it may lie’. One of the most scandalous attempts at the prevention of thought pre-AGW was against Velikovsky. His fundamental notion is that the history of the earth is governed by catastrophic, not uniform events, as Darwin had argued. Accordingly – for Darwin is a religious icon to many – , and as we have seen with AGW, the Guardians moved in to shut him up. A typical weasel defence of this silencing was made by Bauer, summing up the affair. He tells us: Velikovsky’s claims were a priori invalid! Unfortunately, he does not say how he knows this independently of any argument from the ‘scientific’ authority he himself claims. Yet Velikovsky nonetheless obeyed the first command of science: He made his ideas available for public criticism .
Peer review maybe a handy tool for inspecting the depth of research/logical structure of the hypotheses, including possible tests (etc.). But AGW has cruelly exposed its limits. So, provided expressed ideas reach a certain standard of overall literacy (including logical cohesion), anyone is entitled to say anything. The only proviso is they are subject to criticism, the motor of all knowledge. Of course, ideally, the ideas should be new in some sense – scientific discovery is a creative act,and nothing is worse than boredom. It is the least probable hypothesis which is the most interesting. But: A cat may look at a king. Or a professor. More power to your elbow. In a free discussion, all points-of-view welcomed. Incidentally, this answers the question: Quis custodiet ipsos custodes? We do. All of us who willingly share our ideas.
Interesting how that will work. Inside the glass “expertise wall” around the reviewer site, raucus critiquing goin’ on by designated ‘Sperts. Outside the wall, the Watchers have their own hammer-and-tongs discussion going on. Occasionally producing a big enough “flash” or “boom” to require attention from inside the wall.
But where does each “review” start and end? If the “paper” is approved by the reviewers, then what? Entry into an “official” repository of PDFs??
Ain’t we got fun?!?!
I think while science is timeless there is a different spirit in any age. From this I think there are no problems to science. But in any age and any culture speaking wise people – Sokrates, Jesus, Omar Khayyam *) , G. Bruno, G. Galilei, N. Copernicus – were attacked by self-appointed gatekeepers. Some wise people haven’t speak, only written poems on the Cold Mountain rocks (Han=Shan) or 9 x 9 poems in the last days of life (Lao=Tsu).
I read, the Royal Academy of Science were convinced by Sir Robert Ball that communication with the planet Mars was a physical impossibility, because it would require a flag as large as Ireland, which it would be impossible to wave. That was in 1893.
Brian Josephson, Nobel Prize for Physics 1973, says on his homepage: “I am ‘endorsed’ for quant-ph, but find myself blocked when I try to cross-list a paper to make it visible to subscribers to that area, and to search in that area. My letters to the moderators and to Prof. Ginsparg concerning this blocking, which are ignored.”
I think it is obvious that the spirit of this age is determined by rationalism of the mind and so also in the spirit of science. This spirit mode is very different to other ages when science includes philosophy, astronomy, algebra, theology, medicine, jurisprudence and astrology.
I think the basis to be able to speak words like: ‘Keep standards high’ is the personal standing of single persons – like A. Watts – for the timeless ethical principles of science, and not new aristocratic claims on blocking or filtering so called amateurs.
If any publisher of truth feels pressed, he is not interested in truth, but in victory. But science knows no victory. Science is timeless.
The spirit of the age seems to change or to shift to other modes. Hierarchy structures crash down. New structures will build. Science is timeless.
*) “ I was unable to devote myself to the learning of this algebra and the continued concentration upon it, because of obstacles in the vagaries of time which hindered me; for we have been deprived of all the people of knowledge save for a group, small in number, with many troubles, whose concern in life is to snatch the opportunity, when time is asleep, to devote themselves meanwhile to the investigation and perfection of a science; for the majority of people who imitate philosophers confuse the true with the false, and they do nothing but deceive and pretend knowledge, and they do not use what they know of the sciences except for base and material purposes; and if they see a certain person seeking for the right and preferring the truth, doing his best to refute the false and untrue and leaving aside hypocrisy and deceit, they make a fool of him and mock him. “
Omar Khayyám (1048-1131)
V.
I think that your proposed concept has great merit. It is time for scientific journals to take the next step onto the Internet.
There is one particular practice that I would strongly suggest in addition to what other comments have proposed: I would require the submittal of all raw data, methods, computer code and other supplementary materials at the time of a paper’s submittal.
While making supplementary material available is a stated requirement of many (if not most) journals, few (none?) seem to be enforcing it. To me, this embodies the true measure of a scientist, the willingness to allow others to examine their work for errors, biases, oversights, or just plain stupidity. Without access to the underlying data used to reach a conclusion, how can any other scientist, mathematician, engineer or professional every hope to really evaluate the work?
Thanks for the opportunity to contribute to such a noble effort.
I think there is serious agreement to do it.
So, the big question is how to do it. I suggest we could use the ‘hunt the mammoth’ mentality.
The ‘tribe’ picks a few of the better hunters and maybe a few people who have some related experience, we ‘camouflage’ them by assigning an anonymous review number and send them in to throw spears while we watch and comment. A two tier level, one to choose the champions, and the second is the ones who go in and do the work. Then the first tier ‘cheer’ them on, maybe bringing up other points, and throwing in extra ‘champions’ as needed. Say we find somebody in the tribe who specializes in a certain field that is related, we add them to the mix. That would allow a wide group of reviewers, a larger pool of experience, while at the same time allowing us a few representatives of the group as a whole. Then there is a place where the common folk such as myself can input (tribe level) and maybe even bring up points the champions might not see (coaching from the sidelines) where as the champions, say, Monckton? McIntyre? Willis E? to throw spears as they see fit. Maybe adding a referee, say, Dr. Curry, to oversee, and you have a racous process that would allow a more complete review. We could even have the orginal paper writer input, defending the ‘mammoth’ (that might be a really bad idea though, but it could be fun.)
Another point:
For the initial test run of this or any process, we can practice. We don’t need to use a new, untested paper. We can use an existing paper and test it out. The papers exist, and has comments already. Take one that was reviewed, and review it and see which produces a better result.
Thoughts? Better ideas?
@ferd berple
“WUWT could open up a new standard in Peer Review 2.0 by allowing any and all review that speaks to the methods of the paper, and let the conclusions speak for themselves.”
I believe that this is the approach adopted by the Public Library of Science in evaluating papers for their journal Plos 1. If qualified reviewers judge the methods to be valid and the conclusions follow logically from the results, the paper is published.
But even in the evaluation of methods, which may include complex and controversial statistical and mathematical analyses, there will be sharp disagreements, which means that, ultimately, either someone has to make a judgment on what is fit to publish, or absolutely everything is published, in which case we can be sure that much and probably the vast majority of what is published will be, if not rubbish, then at least of minimal significance, even though the methods and the logic are sound.
The Web, however, could provide a means of sorting the near certainly futile from what might contain the germ of an important idea, or the intimation of a significant fact. For example, software could be devised that would attach something like a Google page rank to every publication, but weighted by time since publication and taking account of reader evaluations.
Reader evaluation could be more sophisticated than Amazon’s book review ratings (one to five stars). For example, the rating of a rater would be weighted according to the number and ranking of the weighter’s own publications.
This would not provide a route to absolute truth, but if might be instructive.
CanSpeccy,
Those concerns could be resolved by having a parallel post where anyone could comment. The Web 2.0 peer review article comments should be limited to actual peers with the requisite qualifications, such as being a previously published author, having a PhD in a related field, etc.
Oops, sorry again. The strain of devising an entire new automatic system of peer review overwhelmed a neural circuit.
That last sentence should have read:
For example, the rating of a rater would be weighted according to the number and ranking of the rater’s own publications.
Who is this Hugh Errors? He sounds like trouble to me.
The key issue in getting proper peer review is as follows:
1. Ensure that the field is not infected with group-think/religious rules. This happens from time to time in some fields. It doesn’t matter how many reviewers you have if all scientists are constrained.
2. Ensure that petty disputes/career struggles don’t impact on reviewing. It was well known in medical research 20 years ago that you had to name certain people who couldn’t be allowed to review your paper – they were submitting similar stuff/you had beaten them to it etc etc. If one in three says no, you’re done for. If it’s one in twenty, chances are you will carry the day.
So in general, the greater the number of reviewers, the bigger the likelihood of good reviewing.
But also, the greater the time spent by reviewers reviewing, which leaves them less time for actually doing research themselves.
Trial and error will bring you to an optimum, I suspect……
@Smokey
“Those concerns could be resolved by having a parallel post where anyone could comment”
Yes, there should be a comment thread associated with each article. Comments might be rated in the same way as the article, based on the author’s rating determined as outlined above.
This would not preclude comment by novices and outsiders, but it would make it possible to see at a glance if there is a consensus among the experts in the field.
The comment thread would also be the place where the author would be expected to rebut significant criticism, thus keeping much if not all of the relevant discussion in one place.
This proposal is definitely worth trying.
Rules on who can review, how many reviewers, how long the process remains open etc need to be established beforehand.
I would suggest that one or two reviewers ought to come from ‘outside’ the field of research of the paper under review. The reason for that is that often there is an ‘in-group’ language used which makes it impossible for those of other disciplines to understand. Thus we have the comments of various scientists outside climate science that they don’t really understand what is going on but trust the climate scientists to be right. That’s why we are where we are …
Anyway, go for it, there are lots of good proposals cas to how to proceed ollected here already.
OTOH, I’ve been considering a web 2.0 peer review experiment of my own. WUWT now has the ability to offer a peer review service for articles and papers. It is a new feature I can activate into WordPress, and would allow comments by invited reviewers to be posted for authors prior to publication, so that articles can be evaluated by a broad base of techical readers prior to publication.
I welcome readers thoughts on this idea. – Anthony
It might be a good idea. It might not. I’m not smart enough to predict the possible unintended consequences. The idea brought to mind this XKCD entry:
http://xkcd.com/927/
Not a perfect fit, but close enough, perhaps.
There again, probably worth trying out to see what happens.
Excellent, Anthony, so go for it! As Tallbloke says, ‘an idea whose time is come’. But… please don’t allow the pedants to clutter your proposal with ‘rules’ as the only rule should be that all participants must conduct themselves with brevity, grace and gentility.
Anthony, as one who has deliberated and written an article for submission to WUWT, then stalled, then started again ad nauseum, I would really welcome such a collaborative review process.
If my ideas are to be published and accepted or reviled, at least I would have the benefit of some learned feedback to at least provide confidence that what I submit is viable.
Wouldn’t it be great to be able to publish something knowing that most major bugs were eliminated (alternative to looking foolish to just a few rather than 100,000,000)
Bring it on I say
Warmest Regards
Andy
Start with invited reviewers, but allow the paper and the review discussion to be redable by all. Allow evryone to comment, but comments other than by the reviewers are held until approval. Allow a moderator or a subset of the invited reviewers to promote a comment to visible or to promote a comment author to be an aditional reviewer.
Also, requires that all data and software be available on the web before review starts and eplicitly ask that the reviewers to check that the posted materials are adequate.
I’ve been a reviewer for professional journals, and I have also had to endure review–sometimes enormously useful, sometimes irritatingly self serving. So I think I can speak on a few topics. I know many on this thread would hate to emulate the current system, but some aspects of it are unavoidable.
Journals usually have a chief editor and an editorial board. The editorial board are drawn from the ranks of experts over the range of topics the journal covers. When a paper arrives it is turned over to the appropriate editorial board member who then solicits help from people he/she knows to be area experts that are current with the topics and methods of the paper. The reviewers in turn should voluntarily decline if they have a conflict of interest, cannot do a timely job, or feel they lack the expertise to do a good job; and they would then suggest an alternate reviewer.
Some journals allow an author to suggest reviewers, and this is one practice I’d avoid.
Anthony, if you wish to pursue this further you probably can put together an editorial board from among the people who regularly visit this site, and from those who are willing to reveal their real names and credentials. People do not need advanced degrees, but they do have to be credible in some way.
Someone suggested a print version, but print is hugely expensive. The journals we think of, Science, Nature, JGR, GRL, etc, have stiff page charges (often waved, but often paid by research grants) that you’d be unlikely to collect. Some journals have advertising, and charge very large subscriptions to libraries and institutions. Print is just a headache.