Quote of the week – solving the peer review integrity issue

qotw_cropped

A poll follows.

Over at Bishop Hill, he’s listed some quotes from Geoffry Boulton on scientific integrity that I found interesting. He writes (with apologies for posting in full, I couldn’t see any way to excerpt this short article):

==============================================================

Geoffrey Boulton is giving a speech to JISC, the goverment body which “inspires UK colleges and universities in the innovative use of digital technologies, helping to maintain the UK’s position as a global leader in education”. (Austerity, what austerity?). His comments are being widely tweeted under the hashtag #jiscmrd. Here are a few interesting ones: 

Bolton from Royal Society saying that its “malpractice” to not publish underlying data to research at same time as paper published

@ScienceBL: Boulton says publishing of data should be concurrent with the paper. #jiscmrd #datacite” <- very much agreed.

Boulton: cures for scientific fraud: open data for replication, transparent peer review, personal and system integrity #jiscmrd

#jiscmrd Geoffrey Boulton: open data is our responsibility to citizen science.

It’s funny to see Boulton calling for transparent peer review after failing to investigate allegations of journal nobbling – probably the single most important issue to have emerged from Climategate – during the Russell inquiry.

===============================================================

The idea of having all the data and methods up front ahead of time make a lot of sense. In my view, this is central to effective peer review. Without it, it boils down to a “trust us” situation with the authors of the paper. Given all of the mess surrounding Marcott et al and the failures of peer review to catch its problems, and the uptrend in paper retraction in science in general, I thought it might be a good time to ask this question.

UPDATE: Some people wondered about whether they could join such a professional organization or not if it existed, not being accredited in the field. It should be noted organizations like the AGU and the AMS accept “associate members” i.e. people that have an interest in the science but who may not be accredited in the field. There’s no reason to consider why that could not be the case for a new organization. – Anthony

Advertisements

  Subscribe  
newest oldest most voted
Notify of
knr

Boulton is to scientific integrity what Charles Manson is to happy families.

“Western populations would accept serfdom if it is packaged as saving the Earth”…Bertram Russell
Reject the hypothesis that humans CAN save the Earth and you are on your way to avoiding serfdom….meanwhile we CAN reduce our minor negative impacts. Thinking humans make lousy slaves.

geologyjim

Well, this is truly rich. These statements may set a new standard in “hubris”.
McIntyre, rightly admired for his considerable restraint in dealing with pompous buffoons, must be experiencing a rise in his annoy-o-meter.
This must be the face of post-normal science – where facts, truth, integrity, and logic “don’t really matter anymore” (TM – Climate Science)

Owen in GA

My specialty isn’t atmospheric sciences, so I am not sure I could check “Yes” on the poll. Replace that with RF Propagation and I’d definitely join. The problem is that while I follow this issue and it impacts tangentially on my main research in RF propagation, it is the politics of it that inspired me to look into it farther. When someone claims that we all have to live as cavemen to save the planet, I want to check their data. When the data (when one can get it) and procedures (again if one can get it) appears to not support their conclusions I get curious and start digging. When the further I dig the less supported the position appears, I tend to get angry at those who are muddying up the reputations of all scientists.

pottereaton

Is the poll for people working in the sciences only? I’m not in that group, so rather than skew the results by voting “no,” I’ll refrain from voting. If I were a scientist, yes, I would join.
REPLY: actually, organizations like the AGU and the AMS accept “associate members” i.e. people that have an interest in the science but who may not be accredited in the field. There’s no reason to consider why that could not be the case for a new organization – Anthony

Gary Pearse

Engineers generally have high integrity because the proof of their good or bad work is in the product. You can’t get away with cooking a design for a bridge or airplane, nor would one want to. It also helps that they belong to associations created by statutes that require a license to practice which requires certification, passage of an ethics examination and which has the power to discipline up to and including loss of license, remedial education and supervised re-entry into practice. I think the time is ripe, in an era of moral laxity, to require similar statutes for scientists. In Canada, we have already included geologists and geophysicists in these associations. Re Boulton and posting data. Lets hope its ‘data, the whole data and nothing but the data’. We have seen posting of data in the Shakun paper that is tailor made for it and different from that of his just completed thesis on the subject with no additional research done. Boulton would (from his past partisan position with climate felons) have been a candidate for re-education if he were an engineer.

Come on, you yes-voters, what does the quibble “professional” mean but authoritarianism and credentialism? How many vote is not so significant as who counts the enfranchised-votes. Professional democracy is not different from professional mob-rule. The Ancient Greeks mooted elitism through Sortition.

JC

If I voted Yes it would really indicate something between “Confirmation bias” and “Conventional Wisdom”. If I voted No it would not mean I didn’t want more transparency but simply that I didn’t have the interest or knowledge to contribute.
Then again, like one of the Marx Bros said, it might indicate that I wouldn’t want to join any club that would accept me.
JC

Josh C

I would welcome any scientific journal that had that requirement – I would humbly suggest it would make a great standard for a scientific magazine, online or otherwise. In time I would think it would be far more popular due to a history of integrity that some would say is lacking in the current offerings.

Elsa

I think his suggestion is a start. But it does not deal with one of the erroneous methods used by the warmist lobby which now seems to regard the number of “peer reviewed” papers as being critical to an author’s credentials and correctness. This has been taken a step further by suggesting that because a large number of such papers support the AGW view it naturally follows that there is a consensus that the AGW belief is right and that there are only a few cranks, who should be disregarded, who disagree.
A peer review by somebody who agrees with your point of view is unlikely to advance things very much. It is a bit like asking a group of church ministers to review papers on the existence of god and concluding, because they give favorable reviews, that god therefore exists. Then, because only true believers are likely to publish such papers, you say that the overwhelming number of papers support the existence of god, therefore there is a consensus that he exists.

Boulton is an international expert on ethics. Here, for example:
http://www.raeng.org.uk/news/publications/list/reports/Ethics_transcripts.pdf
…where Boulton discusses the importance of acting with “honesty and care, not committing plagiarism and declaring conflicts of interest” (much like when Boulton was presented as an IPCC author in his resume, as the UK Government’s Chief Advisor on Climate Change (http://youtu.be/2VFWYfBEtJ8) (which he was not), and when he neglected to mention his time at the UEA when serving as a lead in the Muir Russell ‘independent’ inquiry into the CRU. Who better to comment on matters relating to honest peer review? (ho ho)

michael hart

If a professional atmospheric sciences organization existed that didn’t enforce it’s declared standards, would you resign?

I strongly recommend the process of open and transparent peer review as I describe in the following (in the case of publicaly funded research).
Peer review should be performed as it is currently done in a private environment but then the proceeding of the peer review should be made accessible to the public when the paper is published. The reviewers should also be identified as a mandatory part of the public access to the peer review proceedings.
I would not consider incremental cost to do that as a a negative attribute.
John

jayhd

It’s been noted in previous posts, but I will repeat it. Geoffrey Boulton has absolutely no credibility with regards to ethics and integrity. The Muir Russell inquiry proves that. Requiring the publishing of data with the research paper to facilitate replication and open peer review is a great idea, but Boulton is absolutely the wrong messenger.

David Schofield

I don’t twit myself, but is anyone going to do the Phil Jones quote about ‘you will only try to find fault with my data so you aren’t having it..’

I would go so far as to say that “fake” should be the default assumption for any paper that does NOT publish its data when it publishes its conclusions.

Kaboom

A case of ” Why do you see the splinter in your brother’s eye but not notice the log in your own eye?”, even though the splinter is a very real problem here.

John Whitman says:
March 26, 2013 at 9:25 am
Peer review should be performed as it is currently done in a private environment but then the proceeding of the peer review should be made accessible to the public when the paper is published. The reviewers should also be identified as a mandatory part of the public access to the peer review proceedings.
If the journal will not cooperate in this regard, the author can simply put the review on his/hers website. Examples here: http://www.leif.org/research/

Reblogged this on Climate Ponderings and commented:
TAKE THE POLL

Duster

John Whitman says:
March 26, 2013 at 9:25 am

I think that no review or evaluation should be anonymous. Not infrequently, applying for a grant, even a very small one, for instance to cover costs of radiocarbon dating, can meet with a completely opaque “not recommended” or similar response with no explanation why. Without an explanation, there can be no path forward if the researcher cannot fund the work. Not infrequently, back stories and grapevine information will often indicate that “x” review the application. He/she – hates the committee chair of that student/automatically – rejects research without an inherent touchy-feely/component – wants the project and has in fact already hijacked the idea. Requiring a clear justification for why a proposal is reject and that the individual objecting put their name on the line seems to be a clear method of forcing a more objective review process.

Jeff L

This approach should be expanded to all peer reviewed science. Although readers here known & are focused on atmospheric science , similar issues exist in other sciences as well. Are more open process would be beneficial to all science.
Then the next big problem to tackle would be grant funding – needs to be double blind. We do these 2 things & we’ll be a long ways toward more credible science.

Given that I already belong to an organization (contract engineering for mining firms) that requires this by law, I have voted yes. I’ve said this before. If climate scientists had to publish under the same laws that geologists do (hmm I can think of one of those!) they would panic. Indeed, if Mann “hid the decline” in a NI-43-101 report he would be subject to prosecution. A geologist would be required to highlight the decline and explain why it wasn’t important rather than hiding it.

Chad

In some cases I run through hundreds of GB of raw data before I come to a simple final plot. I actually don’t store the raw data many times, since it would take too much space to store it in the first place. This would also be the case for DNA studies. While raw data publication should be encouraged, it should not be required.

John Billot

Anthony, how do we know this is not another “Lewcook” production? 😉

Milwaukee Bob

… JISC, the goverment body which ”inspires UK colleges and universities in the innovative use of digital technologies, helping to maintain the UK’s position as a global leader in education”
With all due respect AND apologies to my English friends, especially those in Hartlepool, the UK “needs” a government body to inspire the use of digital technologies in higher education? And IF the UK IS a global leader in education, why are so many Brits coming over to the US to finish their education? Part of being a GREAT, LEADING education system is having the capacity, in space, cost AND imagination, to “educate” a vast number people to whatever level they so desire and can achieve. Secondly, education is NOT just about scoring high on paper tests, in school. THE REAL test is what you are capable of accomplishing (and do accomplish) in THE REAL world once you are out of school, that of course is IF you ever leave school.
And “JUST” publishing the (some) data with/when a paper is submitted is not going to solve the inherent problems with the cloistered per-review system.

JudyW

The UK is absolutely the best at self aggrandizing propaganda.

oldfossil

Perhaps Geoffrey Boulton has experienced a conversion on the road to Damascus.
As a parent of two and grandparent of five I can attest that the best way to reinforce positive behavior is to reward it.

Sundance

Anthony an equally important questions is “Would you see an organization or science journal providing replicable papers up front as better than those that don’t?”

Another “quote of the Climate Gate Cult’s elite:
http://www.wsj.com
March 25, 2013 ” Natural Gas Rocks The Energy World”
by:
David Crane , NDG (Natural Gas Producer)
Tim Rosenzweig (Goldwind Inc.) must be wind energy guy.
Michael Brune (Sierria Club)
Best quote ” by the Sierria Club guy,,
“why spend hundreds of billions of dollars in a “fossil-fuel” infrastructure when
WE KNOW THAT WE HAVE TO DECARBONIZE AS QUICKLY AS POSSBLE.”
He better check things out, he be % carbon too.
Carbon feeds corn , corn feeds him.

jorgekafkazar

Science is about replicability, not hubristic assertion. If it’s not replicable, it’s not science, and no amount of statistical mummery will make it so. It may be religion or politics or pandering, but not science of any sort.

To appreciate the importance of JISC to this story it important to remember that CAGW is relying on a modelling, constructivist, social inquiry approach to education replacing the historical transmission of a body of knowledge approach. Then you simply make catastrophe due to . . . as politically desired part of the software. Visually the student keeps hearing and seeing that there will be a catastrophe and enough repetitions make it a believed “fact.” remember false beliefs are still consequential in altering future behavior.
The Frankfurt School in their Radio Project research in the 30s established that useful fact. It’s in the Primer for all aspiring Statists I believe. Much of the push in digital learning is really to make education vocational. Use of a tool that does the mental work. So a computer as the center of education is a cognitive tool that is actually weakening most students who simply treat the interaction as visual and interactive, not mental as a book or a lecture.
That’s also in the Primer. Use education practices that take away the masses ability to think accurately and abstractly. Seriously. It comes from Soviet psychologists Vygotsky and Luria and especially Leontiev. I actually explained symbolic cultural tools here months ago http://www.invisibleserfscollar.com/creating-new-minds-different-values-equity-in-credentials-can-this-really-lead-us-to-prosperity/ .
Plus there is the fact that to a committed Marxist (and they do exist and they are getting bolder in saying so), ICT is the dream transformative technology that Marx and Engels always dreamed of in their vision of history. The idea that consciousness changes to reflect the dominant technology.
So if education is the vehicle for transformation and CAGW is the excuse as I have been arguing, the Digital Learning push is another vehicle for massive noetic change. At the individual and then society levels. This is not just another speaking engagement. Digital learning and cyberlearning are very connected to CAGW as a means of sociocultural change. CAGW is the reason for the accompanying economic change. Together you get the Total Transformation or an Administered State. Every bureaucrat’s and their cronies dream.

Ten Commandments of the Climate Cult.
1. Thou shall be cold if the cult says so.
2. Thou shall be hot if the cult says so.
3. Thou shall not eat any thing on our no eat list. (We make the list)
4. Thou shall walk or ride a donkey or horse to work and play.
5. Thous shall live in a small hut and be grateful for that to the Climate Gods.
6. Thou shall not take Michael Manns name in vain.
7. Thou shall not praise the use the Carbon Energy of Mother earth.
8. Thou shall not demand proven facts from the climate gods just obey.
9. Thou shall not do unto the climate gods as they do unto you.
10. THOU SHALL DECARBONIZE OR ELSE.

Theo Goodwin

Gary Pearse says:
March 26, 2013 at 8:40 am
“Engineers generally have high integrity because the proof of their good or bad work is in the product.”
Also, many engineers belong to professional organizations of engineers that teach ethics, have ethics boards, and have something to say about licensing. The standards for Civil Engineers, Materials Engineers, and quite a few others are very high. Most important of all, these engineers are quite comfortable talking to one another about ethics.

Wamron

Re Joseph Oulson: “Thinking humans make lousy slaves”
Not if the Ottoman empire was anything to go by. Their entire state beurocracy was administered by slaves, up to and including the status of regional governor. These were very educated , thinking slaves. Most were raised from childhood for their roles having been taken as human tax tribute from East European states defeated in slave yielding military campaigns. Those campaigns in turn waged by slave armies, most notably the elite Janisserie and the Horsemen of The Porch.
Chattel slavery continued in Turkey until the 20th Century. The last slave market in Istanbul was in 1904.
The vision of slavery that guilt-whipped Westerners are perpetually fed is very, very distorted.

The thought has just occurred to me that more people read this blog than anything Boulton writes.
Indeed, after thinking: “they will soon start coming to you begging to be allowed to speak”.
I began wondering just how much commercial value there is in WUWT.

The climate waterheads have turned “peer review” into “pal review”.
The pretense of impartiality still exists, for some unfathomable reason. It doesn’t make bloody sense.
It’s like catching your child with his hand stuck in the cookie jar, chocolate cookie well in hand, wrist stuck in the bottleneck; yet, the child insists that what you’re seeing ISN’T what you’re seeing.
Guys caught in the family bed banging away at a chick have more credibility than Albert “Global warming’s gonna make me rich, beyotch!” Gore.

Dave

Boulton is to scientific integrity as Gliek is to ethics…

William McClenney

Geoffrey Boulton, the poster-child for post-normal science and the Theory of Inverse Reality. Invert what he says and then discern the truth of whatever matter.

Stephen Richards

I wouldn’t join but I would buy their journal. I buy none now. Stopped physics, SCIAM, Nature, New scientist. Had them all at one time.

john robertson

Boulton speaking on ethics and science is like Al Gorical speaking about energy conservations and honesty.

lsvalgaard on March 26, 2013 at 9:42 am
If the journal will not cooperate in this regard, the author can simply put the review on his/hers website. Examples here: http://www.leif.org/research/
– – – – – – –
Leif,
Thanks for your comment.
I noticed your “320 Dikpati Referee Report.pdf (My Referee’s Report on Dikpati et al., GRL 2006) “.
As a referee (reviewer) option, publishing their reviews on a website does look like it contributes toward opening up the journal (peer) review process.
John

Wamron

Milwaukee Bob…all accepted and with no offence taken.
That thing about British education is one of a number of articles of faith that form unexamined wallpaper to life in Britain. Its an identity narrative.
What I really find questionable is the assumption that encouraging the use of digital technology in education is necessarily a good thing.
I very much disagree.

“data for replication”
Why couldn’t he say “Preservation of original data is supposed to be one of the highest priorities for scientists. Preserving data allows others to replicate results, and it also allows for the possibility that in the future scientists and interested people may make observations from that data that are not anticipated by today’s limited understanding. Preservation of unaltered data is a safeguard against many scientific mistakes and abuses, such as carefully selecting only certain data points and signals, while excluding others which would not confirm the the theory of the scientists.” Probably because it was just too long for a Tweet. (:

Duster on March 26, 2013 at 9:47 am

John Whitman says:
March 26, 2013 at 9:25 am

I think that no review or evaluation should be anonymous. Not infrequently, applying for a grant, even a very small one, for instance to cover costs of radiocarbon dating, can meet with a completely opaque “not recommended” or similar response with no explanation why. Without an explanation, there can be no path forward if the researcher cannot fund the work. Not infrequently, back stories and grapevine information will often indicate that “x” review the application. He/she – hates the committee chair of that student/automatically – rejects research without an inherent touchy-feely/component – wants the project and has in fact already hijacked the idea. Requiring a clear justification for why a proposal is reject and that the individual objecting put their name on the line seems to be a clear method of forcing a more objective review process.

– – – – – – –
Duster,
Thanks for your comment.
I think you are suggesting that at a the request of the funding applicant any decision not to fund by the funding body should be explained in at an accessible public link.
It would tend to contribute toward integrity auditing of the funding body.
John

Roy

Milwaukee Bob says:
March 26, 2013 at 10:04 am
And IF the UK IS a global leader in education, why are so many Brits coming over to the US to finish their education?
For much the same reason as American students go to universities in Britain and other countries.

Theo Goodwin

Doug Huffman says:
March 26, 2013 at 8:52 am
Please do not panic. Lots of ideas are floating around here. The idea that data must be published along with the publication of a peer reviewed article does not threaten the liberties of anyone.
As regards professional associations, they are found at various places along a spectrum. An association that teaches ethics and encourages its members to become comfortable discussing ethical issues does not threaten the liberties of anyone.
Keep in mind the context. The so-called “investigations” that white washed the Climategaters could achieve their end only because there were no widely accepted standards that could be applied to their work. In effect, the investigators made up their standards or non-standards during their investigations. That cannot be allowed to stand. The behavior of the Climategaters and those who exonerated them raised the questions that we are discussing.

Puppet_Master_Blaster_Master

The AGU and AMS “may” be ‘professional’ organizations but THEY ARE THE CENTAL PROBLEM with “peer-review”, including their pay-to-play systems and ‘buddy-check’ reviewers.

Certainly the situation as it stands has to turn around or all of science loses out.
I find it personally interesting that while I know it is the scientists amongst us who are doing some heavy duty work to turn this scam around, and it is scientists who really will save this world from mad green policy – I nevertheless have a default reaction of mistrust whenever I hear “science says” now.
I hate that I feel that way. It surprised me that I should have that initial reaction to reject something outright simply because “science says”. It doesn’t matter what the field is. On some kind of subconscious level “Science = Dodgy” and that is just SO WRONG.
I’ve been coming in here daily for quite some time now – I’m addicted to this place. I love what I learn here, I love reading the comments and interacting with you all. I know that we will win out through using real science and the scientific method. You are all doing exactly what’s needed.
Yet there is a huge percentage of the population that does not come in here and does not know the good work being carried out to stop the madness that’s going on in the world. When this is all done and finished, it will be the “skeptical” aspect that will be remembered. “Science” will be remembered as being on the Bad Guy side.
So, if I’m feeling “science = dodgy” and “don’t trust ‘science says’,” and I KNOW that real science is winning – what the heck are THEY going to think? Especially when the only science that’s been rammed down their throats for 30 years has been the FALSE sort.
So, yes, this situation has to be turned around to keep ALL of science going down the toilet completely for the next hudred years.
After my initial mistrust when I hear “science says,” I have only one question and that is, “Where’s the data?”
Data and methods ALWAYS supplied up front is the way to go. It is the vital step. That, I would always trust.

Ray Donahue

Hi Warmron, RE: Slavery
Absolutely correct. A very (!) complicated and embarassing subject (not just to the Western World
Ray

Vincent

Pressure of work prevents me from reading previous posts, so I may be repeating points already made.
The questions being asked in this poll makes it irrelevant for me to respond, but it raises issues that I have a strong opinion on. The problem is the phrasing of the question:
1. I’m an Electronic Engineer, so I would never be likely to join an atmospheric sciences organization.
2. Even if it was an engineering organization, I would not be likely to join either way. After more than 30 years in practice, I’ve not joined one yet.
However I do read papers from these organizations and I need to trust what I read.
What is perhaps more relevant to this poll is if it had asked whether I would give credence to a paper published by an organization that only published papers that were replicable up front.
Then my answer would be a definite “Yes”! As it stands, I can’t even say “Maybe”.