what’s the point of peer review, anyway?

July 2, 2007 at 3:32 pm | | literature, science community

I’m pissed off.

My PI recently asked me to help him referee a manuscript (obviously the authors and journal will remain unnamed). I read the paper carefully and wrote a detailed report, requesting several changes before being appropriate for publication.

This manuscript had many major issues (including reproduced but uncited figures, inconsistent structures, poor statistics, and claims of data in the abstract that were nowhere in the paper). My referee report was friendly but asked for corrections. I was even kind enough to split the comments into major and minor.

We never heard anything back from the editors of the journal after we submitted our referee report.

So I was shocked to see the paper published recently … without any of the changes we requested! The reproduced figure—identical to an earlier publication—remains parading as original data, the structures remain inconsistent, the extraneous claims remain in the abstract, the statistics remain unjustifiable. WTF? Maybe two other referees said to publish without changes, but my comments should have been helpful to the authors and editors! I was trying to make the article better and help the authors emphasize their results. What was the point of all my work?

[Update] I think that the editors must not have sent on or looked at my referee report, because the authors didn’t even correct a wrong page number on one of their references. What jerks!

11 Comments »

RSS feed for comments on this post. TrackBack URI

  1. That’s horrible, and insulting! A friend of mine recently spent time to construct a thoughtful review of a paper he thought had fundamental flaws only to find out that it was published without addressing his comments. He e-mailed the editor who told him that he was on the authors’ list of people to exclude as referees. He kind of apologized and said that happens a few times a year. The funny part is that the editor said he also sometimes sends a paper to be reviewed by one of its authors!

    Comment by Andre — July 2, 2007 #

  2. that’s annoying, andre. there’s no way we’d (legitimately) be on the list of people to exclude from referees, though: we don’t do any overlapping work with the authors.
    .
    i wouldn’t mind refereeing my own paper, though.

    Comment by sam — July 2, 2007 #

  3. That’s really terrible. Especially considering the difficulty passing the referees sometimes. And here they totally ignore real, helpful comments. I hope this is an anomaly…

    Comment by Eli — July 3, 2007 #

  4. Yeah, our group just had a situation where the journal proofs for a paper were sent to us BEFORE we had a chance to respond to the referee’s comments. I think the editor’s job in many journals now is more co-ordinator than copy editor and their primary mission is to get the next issue out while assuming that the authors get things right on the first bounce.

    Comment by Liberal Chemist — July 4, 2007 #

  5. That’s pretty crappy, but nobody was claiming peer-review is perfect. Look at the opportunity now: write a rebuttal.

    Put together a “letter” or “note” to the journal, with all of your major criticisms, and send it to the editor for publication. The editor will then contact the authors of that paper and they’ll have a chance to respond publicly to your critique. It’s fair, you get a publication to pad your CV, they can defend themselves, the citation rate for the journal (and original paper) both go up a little, and the broader scientific community gets to see what they did right and what they did wrong.

    It’s also possible that journal will not accept your critique… but you can easily send it to another journal, where the same process will happen.

    This kind of thing has happened to my advisor, too. He showed me one paper that he’d recommended be rejected for a variety of reasons, that got published shortly thereafter anyways. He’s also got a story about seeing a manuscript for review three times for three different journals. He showed me the last review – it included a long, but fairly polite, rant about not just shopping a flawed manuscript around until acceptance.

    Sorry for the long comment, but I think you should put together that critique right away – no point in waiting for the paper to get stale, and you’ve got the review already written, so it should be relatively easy to pull out the major bits and put them into a 500-word essay.

    Comment by TheBrummell — July 5, 2007 #

  6. thanks for that suggestion, TheBrummell. i think that’s a reasonable idea, but none of the problems with a paper really deserve a rebuttal.
    i guess i just feel that i tried to help the authors and editors make it a better paper, and they rejected my help.

    Comment by sam — July 5, 2007 #

  7. “none of the problems with a paper really deserve a rebuttal.”

    um… what about the blatant rip-off figure? OK, maybe they just forgot to put the proper citation into the figure legend, or make it clear that the figure is used (with permission?) from another source. That’s dancing pretty close to copyright infringement or plagiarism, though, to be blunt. The extraneous claims in the abstract are also, in my opinion, sufficient grounds for a short letter – not necessarily a rebutal, but a polite letter drawing the editors’ attention to these problems looks justified.

    “the statistics remain unjustifiable.”

    That also looks like more than sufficient to write, again, a short, polite letter.

    Peer-review doesn’t begin at submission and end at publication – letters, attempts to replicate experimental results, and contradictory results or solid reasoning all count as part of the peer-review process, in my mind. If the gold-standard part of peer-review (the bit that happens between submission and acceptance) fails, then there are still other components of peer-review that can be employed.

    Now it looks like I’m berating you for not insta-publishing a short, sharp, shock of a letter-to-the-editor. Sorry about that. But I still think you should write to the journal.

    Comment by TheBrummell — July 6, 2007 #

  8. i agree with you, TheBrummell. i’ll mention it to my pi: it’s really his call.

    Comment by sam — July 6, 2007 #

  9. One of my paper recieved high marks (7/10) from both referees, yet it was still rejected. Their comments were ambiguoustic and contradictory. In one way, they gave me high marks, and yet on the other hand they do not recommend for publication. So… Anyway, I accepted the feedback and cooments, improve my paper, and try again. My boss, being the honest one, told the editor in the cover letter, that it has been rejected once and this is the improved version. The editor didn’t even bother to send it to the referees. The manuscript was sent back to us 2 days later. What do you think? What should you do in such situation?

    Comment by taitauwai — July 9, 2007 #

  10. man, taitauwai, that’s annoying. it sounds like you’ll have to submit it to a different journal, but maybe one of equal caliber? unless your PI is willing to fight the editors on this one.

    Comment by sam — July 9, 2007 #

  11. […] happy that the editors sent us these referee comments, so I can fix things. (I wish the editors of another journal had sent my referee notes to the authors!) Nevertheless, maybe the editors should have waited an extra few weeks before telling us to start […]

    Pingback by Everyday Scientist » late referee reports — October 21, 2008 #

Leave a comment

thanks for the comment

Powered by WordPress, Theme Based on "Pool" by Borja Fernandez
Entries and comments feeds. Valid XHTML and CSS.
^Top^