nature’s peer review trial

June 8, 2006 at 10:47 pm | | literature, news, science and the public, wild web

The journal Nature has decided to perform a new trial peer review process. Check it out here. During a trial period, authors who submit articles to the journal for publication can opt to have their manuscripts available on an open forum, where the whole world can read and comment on the preprint. The normal review process would occur in parallel, but the editors (and presumably reviewers) can read the public comments and take those into consideration in their decision to publish.

I think this is sorta weird. I mean, do we really want the entire world reading our preprints before they are accepted? I mean, there’s a reason we try to keep the entire public out of the loop a little: there are a lot of crazies out there! For instance, I don’t really know anything about Ivor Catt, but he sure seems crazy. Well, I guess it will be sorta like arXiv. I’m not completely opposed to the idea, I just don’t think it’s the best first step in correcting the peer review process.

The first thing I would do is change the half-blind style of reviewing: reviewers know who wrote the paper they are reviewing, but the author never knows who accepted or rejected their manuscript. This is silly. Reviewing should either be fully double-blind or fully open. I lean toward double-blind, anonymous reviewing: I can’t think of any good reason either the reviewer or the author should know the name of the other. The alternative (double-unblind?) would mean more accountability, but might encourage personal vendettas. The current scheme means that the reviewer isn’t accountable—and can either let vendettas fly or let a manuscript slide in by the merits of the author.

Well, I admit that I haven’t really thought about this enough. There’s a lot more to discuss on this topic, but maybe that could be for another post, another day. Or we could head over to the Nature discussion forum on peer review.


RSS feed for comments on this post. TrackBack URI

  1. The site diplays articles in blog format. And since I didn’t stumble across this myself but read it from Everyday Scientist you can see their post about it from here Finally, since Nature is posting preprints in blog format, we would be able to add these preprints to our forum feed and they would be made more visible in case anyone wanted to comment on them. If you want them added to our forum feed please post

    Pingback by Chemical Forums Blog - A Chemistry Blog — November 18, 2006 #

  2. Hello Sam, thank you very much for the trackback and post on our peer review debate.
    I thought that comment by Ivor Catt kind of proves the point about the value of peer review rather neatly;-)

    On your double-blind comment, I agree that a lot of people (scientists) instinctively believe this is the fairest way. If we get a lot of people making this suggestion in the comments to our debate, who knows? But I would just say that as part of our interview procedure for editors, we provide an “anonymised” manuscript for the candidate to assess, and without exception they know the author straight away (not least becuase the author ususally cites his/her own work in the first few refs, but also because in most fields people know who is doing what).

    I’m not disagreeing with you, just pointing out some experience.

    The debate will have five new articles added weekly through June, so I hope you and your colleagues will come over and comment.

    Incidentally, have you seen another Nature blog, the sceptical chymist? (Don’t ask me about the y, I don’t know).

    All the best and thank you again

    Comment by Maxine — June 9, 2006 #

  3. Thanks for the comments, Maxine. I know that most scientists can tell who wrote a paper, but that doesn’t mean that we should tell them when they don’t know. :)

    Comment by sam — June 9, 2006 #

  4. –what would be the harm in letting crazies see pre-prints? Just ignore them. The benefit might be a few good comments here and there, and perhaps heightened anticipation of the next issue of Nature.

    –what would be the harm in making submissions anonymous? From a science standpoint, I can’t see one. From a marketing standpoint, it might be problematic. You want reputable names, names associated with reputable institutions or ideas, and not any irreuptable names or institutions.

    Still, if the review were based on a rubric, then perhaps a few reviewers could be responsible for assessing name recognition, independent from the substance of the submission. That assessment for name recognition could be translated into some ranking or score, and the actual paper could then be independently reviewed without the name attached. This, of course, would do little for those who, by virtue of background information, can link the work to its author (Maxine’s point), but for those without the background info, this seems worthwhile (Sam’s point). I suppose the drawback here is that it’s regressive against unknown authors. But maybe that’s a good thing?

    –What would be the harm in making reviewers known? I guess there’s a fear about personal vendetta against reviewers who don’t like an author’s paper. There’s an idea that they should be free to evaluate without fear of reprisal. But if they know whose work they’re reviewing, it’s they who might be doing the reprising. To temper this potential problem, it seems sensible to make the reviewer known, or at least accountable.

    Similar to what I wrote above, candidates for publication could evaluate reviewers on a number of the reviewers’ qualities, according to some rubric. Candidates could submit this review before any final decisions are made. It could be translated into some rank or score, and weighed for/against that reviewer’s suggestions. Perhaps a review could have a rank or score averaged from multiple candidates’ reviews over time.

    –And maybe I’m way off base. I’m more familiar with the process of peer reviewed law journals, so my ideas might be wholly inappropriate for here. In any case, I’m mostly thinking out loud, procrastinating (and feeling very guilty about it) from studying for the evil and imminent bar exam. And now it’s time to study.

    Comment by jordan — June 12, 2006 #

  5. […] nature’s peer review trial from Everyday Scientist The journal Nature has decided to perform a new trial peer review process. […]

    Pingback by Peer-to-Peer: Quality and value: How can we research peer review? — June 12, 2007 #

Leave a comment

thanks for the comment

Powered by WordPress, Theme Based on "Pool" by Borja Fernandez
Entries and comments feeds. Valid XHTML and CSS.