As readers of this blog and others such as Climate Audit know, reviewing new ideas, essays, and papers can often progress very quickly with the help of a widely varied readership. But we still wait for traditional methods somethimes, and for the fast pace of venues like this, it can be excruciatingly slow.
A good example is the AIRS satellite data which promised to give us the first satellite derived CO2 measurement with higher resolution and greater global representativity than the land based measurements such as that from the Mauna Loa Observatory. My writeup on AIRS here on July 31st 2008 indicated that they would have a published paper in 6-8 weeks for us to review. It was a molasses-like wait for many, and I got many inquiries about it during that time. Finally, on September 29th, the paper was published, and I featured it here. That 8 weeks was long by electronic media standards, but pretty quick by traditional science journal standards. But that may be about to change.
The idea of peer review via web collaboration seems to be catching on and this article below demonstrates. – Anthony
From the Economist, September 20th, 2008 (h/t to Dave Stealy)
IN PRE-INTERNET times, peer-reviewed journals were the best way to disseminate research to a broad audience. Even today, editors and reviewers cherry-pick papers deemed the most revelatory and dispatch them to interested subscribers worldwide. The process is cumbersome and expensive, but it has allowed experts to keep track of the most prominent developments in their respective fields.
Peer-review possesses other merits, the foremost being the ability to filter out dross. But alacrity is not its strong suit. With luck a paper will be published several months after being submitted; many languish for over a year because of bans on multiple submissions. This hampers scientific progress, especially in nascent fields where new discoveries abound. When a paper does get published, the easiest way to debate it is to submit another paper, with all the tedium that entails.
Now change is afoot. Earlier this month Seed Media Group, a firm based in New York, launched the latest version of Research Blogging, a website which acts as a hub for scientists to discuss peer-reviewed science. Such discussions, the internet-era equivalent of the journal club, have hitherto been strewn across the web, making them hard to find, navigate and follow. The new portal provides users with tools to label blog posts about particular pieces of research, which are then aggregated, indexed and made available online.
Although Web 2.0, with its emphasis on user-generated content, has been derided as a commercial cul-de-sac, it may prove to be a path to speedier scientific advancement. According to Adam Bly, Seed’s founder, internet-aided interdisciplinarity and globalisation, coupled with a generational shift, portend a great revolution. His optimism stems in large part from the fact that the new technologies are no mere newfangled gimmicks, but spring from a desire for timely peer review.
However, what Dr Bly calls Science 2.0 has drawbacks. Jennifer Rohn, a biologist at University College London and a prolific blogger, says there is a risk that rivals will see how your work unfolds and pip you to the post in being first to publish. Blogging is all well and good for tenured staff but lower down in the academic hierarchy it is still publish or perish, she laments.
To help avoid such incidents Research Blogging allows users to tag blog posts with metadata, information about the post’s author and history. This enables priority of publication to be established, something else peer-reviewed journals have long touted as their virtue.
Coming home to roost
With the technology in place, scientists face a chicken-and-egg conundrum. In order that blogging can become a respected academic medium it needs to be recognised by the upper echelons of the scientific establishment. But leading scientists are unlikely to take it up until it achieves respectability. Efforts are under way to change this. Nature Network, an online science community linked to Nature, a long-established science journal, has announced a competition to encourage blogging among tenured staff. The winner will be whoever gets the most senior faculty member to blog. Their musings will be published in the Open Laboratory, a printed compilation of the best science writing on blogs. As an added incentive, both blogger and persuader will get to visit the Science Foo camp, an annual boffins’ jamboree in Mountain View, California.
By itself this is unlikely to bring an overhaul of scientific publishing. Dr Bly points to a paradox: the internet was created for and by scientists, yet they have been slow to embrace its more useful features. Nevertheless, serious science-blogging is on the rise. The Seed state of science report, to be published later this autumn, found that 35% of researchers surveyed say they use blogs. This figure may seem underwhelming, but it was almost nought just a few years ago. Once the legion of science bloggers reaches a critical threshold, the poultry problem will look paltry.
I think there may be resistance. This good change will come but it may take while. Lots of money and jobs tied to the status quo.
Good site. Index works well. I think it will be a hit. Current peer review has lost it’s authority. Sign of the times.
Traditional peer review died the day the Lancet published a paper on six hundred and fifty five thousand Iraqi dead….. Pure activism disguised as statistics and science…. and nothing more.
http://americandigest.org/mt-archives/tinfoil_brigade/lancet_iraq_dea.php
Those of us who grew up with Usenet and NNTP news-clients find Web 2.0 a pretty inadequate substitute for a newsgroup. Apart from the lousy ergonomics of trying to use a web-browser to navigate a blog-based discussion, there is the truly massive problem of not being able to fork discussions. If you’ve never been able to do it then maybe blog feedback seems like a giant leap forward, but it actually nips debate almost in the bud. I can easily compare the depth and range and vigour of the discussion on one newsgroup is still follow, with the brief and superficial discussion I see on a blog that addresses the same community. The web is like a glossy magazine: slick but thin. Web 2.0 is not the best we could do. It isn’t even as good as what we already had.
A very good point.
My articles at CO2sceptics.com have only been published via the web and on the basis of the qualifications and experience of all the commentators I’m satisfied that I am getting a pretty thorough peer review especially from those who disagree.
Plenty of minor criticisms have some validity but nothing yet to discredit the main gist of what I say and a lot of support from many knowledgeable individuals.
The BMJ have been switching to this type of publishing. Their articles are published continuously, though still grouped into a print issue. All articles allow instant feedback with the best selected as “letters” for the print issue and for author feedback. They also have reader polls and bloggers.
I think this concept is better than the broken peer-review gatekeepers and the well documented fraud.
One needs be cautious he remains discriminatory in what and he reads. Increase of information means we have so much to cover and may read more superficially. It may also be possible that with so much research on one side of an issue that people read solely their team and exhibit confirmation bias.
The advantages I see to crowd-sourcing peer review are 1) raising the likelihood of getting valid criticisms, 2) acquiring new collaborators, and 3) improving the research before it has progressed very far. What gets traded off is the credit for your work. It won’t be harder to figure out who thought of what (the audit trail is on the web), but the credit will be more widely shared (between me and my 200 “co-authors” in the blogosphere). In terms of science advancing, this is a good thing. In terms of careers advancing and getting grant funding, maybe not so much. Grant funding probably will go even more for data acquisition and less for final reports. Software advances should reduce a lot of the administrative overhead, anyway.
Lame duck Gordon Brown is about to shoot the UK in the head, based on the advice of Hansen and Gore.
http://www.guardian.co.uk/environment/2008/oct/07/carbon.emissions.targets
J.Hansford. (00:55:29) wrote: “Traditional peer review died the day the Lancet published a paper on six hundred and fifty five thousand Iraqi dead…….
http://americandigest.org/mt-archives/tinfoil_brigade/lancet_iraq_dea.php”
The link doesn’t work.
Jack Koenig, Editor
The Mysterious Climate Project
http://www.climateclinic.com
Talking of which, are September’s temp numbers out yet?
Roy,
I think this development is promising if unfulfilled as yet. A lot of people have pooh-poohed ideas that worked which at the time were in their infancy. History is littered with them. It may not be a comprehensive immediate improvement, but it does provide an avenue for that over time. With any revolution its usefulness with prove or disprove itself with the passage of time. Even the current state of blogging needs to undergo some evolution. But that takes a broad spectrum of cooperation on a scale we have not yet seen. If it is properly done, it has the potential for fantastic success on a wide array of fronts. And it may lead to revolutions, if it becomes popular enough, in web browsing and computing in general for popular consumption.
One thing I can say for certain, if people see the need for an improvement, they will do it. If a USENET-style of operation provides some advantage, it will be integrated. For now, however, we are talking merely on a scientific plane. This will be much like an online journal (I read National Review for instance) but with the more free structure of a blog, yet it sounds as if they may organize it like a BBS. We shall see. But the greater exposure and interaction of ideas has never, to my knowledge, been a bad idea. That principle has brought us to where we are now.
Nice observations, Gary. I quite agree. Changes are in the wind.
“1) raising the likelihood of getting valid criticisms,”
Indeed. An example, Charles Johnson’s comparison of the Burton Word doc “typed” on an IBM Selectric; his photoshop autopsies of AP pictures of Beriut burning, etc., seem to indicate an adjustment to Churchill’s metaphor, to wit:
“Rumour and falsehood travel around the globe before Truth get’s its pants on.”
Science defined by scientists and administered by journalists securing the remuneration of both is something akin to ‘Government of hens, by the foxes, for the benefit of foxes’.
This raises the possibility for greater transparency. A theoretical scientist can post a paper on line and have it open to critique by anyone, from fellow scientists in his field, to other scientists in related fields, to engineers, even to scientifically literate laymen. The evolution of this approach should be fascinating to see over the coming years.
The current peer review process, hide the data and publish political science, is lacking in many ways. It’s quite possible that forums, where there is open discussion, will force the data into the open sooner, but my guess is there is a certain segment of science which will resist to the bitter end.
And I think you know who these ‘scientists’ are, and what international organization they are associated with. It’s sad, because if the pursuit is ultimate truth, the open science way is best — “The Wisdom of the Crowd” is the best way to engage humanity in any pursuit.
In the end, it’s probably blogs like this one, and others, which will lead the way into the future, and pursue real science.
‘Government of hens, by the foxes, for the benefit of foxes’ — Very appropiate.
This reminds me — next week we should see if the debate on global warming continues at one of the APS newsletters (15 Oct?):
http://www.aps.org/units/fps/newsletters/index.cfm
I wonder who’ll be battling it out this time around.
Love the net. Love having access to data. In Oregon, there are a bunch of us that usually responds to brash statements by saying, “prove it”! So when someone says that global warming is causing drought here, I only have to go to this site:
ftp://ftp.wcc.nrcs.usda.gov/data/snow/update/or.txt
and look up the total precipitation % of average for the month. This is a piece of data that looks at the month so far and how it compares to the average for these collection sites for precipitation of any kind that can either be measured directly, or melted then measured. So far, in Oregon basins, not a single site is below 183% of average. The top performer is the Coast Range with 600% of average. And interestingly, a dry spot in Oregon; Harney; is coming in at 575% of average for October. So something is seeding the clouds cuz we are getting DUMPED on!!!!!!
And along with the above data, it is now nearly 10 degrees colder here in Enterprise than it was last year at this time of the evening. Most of the area I call home is scheduled for freezing temperatures and snow. An event that is coming earlier than last year (and last year we thought it was early).
Roy (01:04:44) :
> Those of us who grew up with Usenet and NNTP news-clients find Web
> 2.0 a pretty inadequate substitute for a newsgroup. Apart from the
> lousy ergonomics of trying to use a web-browser to navigate a
> blog-based discussion,
or copying text between text area and real editor and back when
adding to a comment. (Mostly fixed by a Firefox plugin called
“It’s All Text!”)
> there is the truly massive problem of not
> being able to fork discussions.
Perhaps one thing that could be done is to create a new “Off topic”
post each week as a breeding ground for new articles.
Another problem is I wold read a News thread until replies got
5-6 deep, by then the topic would be lost, the detail too nitty-gritty,
or lost in a flame war. Any new offshoots would have a new
subject line. Doesn’t happen on a forum like this.
> … I can easily compare the
> depth and range and vigour of the discussion on one newsgroup is [I?]
> still follow, with the brief and superficial discussion I see on a
> blog that addresses the same community.
This blog and many others like climate audit are okay. A lot,
especially news media blogs, seem to turn into teenage-instant messaging
or cell phone text messages. I hope we aren’t raising a generation that
can only think in sound bites or words with single character
abbreviations.
> … The web is like a glossy
> magazine: slick but thin. Web 2.0 is not the best we could do. It
> isn’t even as good as what we already had.
Unfortunately USENET was taken over by the spammers after moderated
groups had picked up an ugly stigma. Imagine how bad blogs could be,
and the current dysfunction looks pretty good.
Ah, it’s so nice using “proper” quoting style instead of HTML. 🙂
At least, assuming this formats okay….
Debate? From A Tutorial on the Basic Physics of Climate Change:
Further:
Fascinating and timely.
Gary: “The advantages I see to crowd-sourcing peer review are 1) raising the likelihood of getting valid criticisms, 2) acquiring new collaborators, and 3) improving the research before it has progressed very far. What gets traded off is the credit for your work… In terms of science advancing, this is a good thing.” Absolutely.
Coincidentally, right now I’m using my own resources (phpBB forum plus website) to develop the same idea of collaborative online research. I’m endeavouring to start to knit together and clarify the climate skeptics’ positions, to help develop an agreed body of skeptical science that can rebuild the real Climate Science. I started with writing a primer, then I could see my own limitations – we need (a) enough agreement on issues, in order to (b) have statements that refute all the AGW straw-man refutations which constitute their fuel lines.
I’d ask Anthony to run a piece on this work – but am not sure yet about coping with the mass influx that might create! However, do have a look HERE, since it is all interesting work in progress… and hopefully will help reclaim Climate Science…
Web 2.0? I haven’t finished the first one yet…..
Hi Anthony,
I’m teaching a course on Web 2.0 (for college teachers) here in Quebec and I’d like to use your image (web 1.0 to Web 2.0) for it. I would translate it in French and post it on my course. I would give this webpage (link) as the place where i got the original. Is it ok?
Raymond