broken peer review

Sep 11 2011 Published by under Uncategorized

OMG academic peer review is totally fucked and it never works and the third reviewer is a stupid idiot asshole!!!11!!!1!

I am certainly guilty of complaining about the review process when my (uniformly awesome) work is the target. Today on twitter I learned that Leonid Kruglyak (@leonidkruglyak) and I have something in common in this regard. He complained:

Obviously, I don't know what the reviewer actually asked for*. And, though I don't often want to admit it, if I'm being honest, most** of the comments from reviewers do end up making the work better. The link in that tweet goes to a column by Hidde Ploegh published a while ago by Nature arguing that the peer review process is a "wasteful tyranny" and that it is worse the higher-up the journal food-chain you are. In this column, he proposes 3 steps to improve peer review:

First, they should insist that reviewers provide a rough estimate of the anticipated extra cost (in real currency) and effort associated with experiments they request. This is not unlike what all researchers are typically asked to provide in grant applications. Second, journals should get academic editors with expertise in the subject to take a hard look at whether the requests of reviewers will affect the authors' conclusions, and whether they can be implemented without undue delay. Third, reviewers should give a simple yes or no vote on the manuscript under scrutiny, barring fatal shortcomings in logic or execution.

I might be mistaken, but it seems to me that these steps would just push all the blame to the editors. I have not seen evidence that the peer review process is much different at places that use academic editors as compared to professional editors.

I think that part of the reason we all complain about peer review is that our papers get rejected. And this isn't going to change any time soon. Most journals, especially the "high-impact" magz, get far more submissions than they can publish. That being said, all of us that complain about peer review are the people we complain about. I am sure that there are times that folks would characterize me as a "third reviewer". I am not an evil asshole, trying to scoop a competitor, or lazy***. My goal is always to write a logic-based, thoughful review that I would be happy not be pissed off to read as an author. Do I make mistakes? Certainly. But I am trying.

In the end, the only way that peer review will get better is if we "peers" decide to change it. If there are established folks out there (like Drs. Plough and Kruglyak) that act like the "perfect" reviewer, and train the people in their labs to do the same, maybe the cycle of "look, I've read it, I can be as critical as the next dude and ask for something that's not yet in the manuscript" can end.

It will still suck if my paper gets rejected.

_________

*IME, the author's paraphrasing of reviewer requests can be a little hyperbolic.

**I am not arguing that there are exceptions or that EVERY reviewer comment is worthwhile.

***Though these things may happen with some reviewers, I think that they are probably the exceptions, not the rule. Call me an idealistic dim-wit.

23 responses so far

  • Martini says:

    In my experience it is pretty straight forward to rebut a reviewer that asks for a new shiny expensive experiment that adds no substantial data to the main conclusion(s) of a paper. Especially if you lay out the time and costs to the editor. These are also the easiest reviews to rebut because once you have done 2 or 3 simple experiments that prove a point, it is unnecessary to do the big shiny expensive experiment to basically prove the same point.

    The reviewers may have a point though if you were indeed missing a control or overlooked obvious alternate explanations.

    However, some reviewers will use the sorts of things as an easy way to justify rejection of a paper. In this case, they weren't going to give you favorable review no matter if you had already done that new shiny expensive experiment.

  • CoR says:

    I've had a couple of really terrible reviewers. On the whole, however, I have had a very supportive bunch--not that they lacked criticisms, but that for the most part reviewers have given me good suggestions. I have noticed that the bigger the paper in terms of IF, the better the reviewers and their comments.

  • Yoder says:

    I will say that the nastiest, most baseless* review I've received, in my brief career, was at the highest-ranking journal I've submitted to. But there was a heckuva lot at stake, so I sort of figured it comes with the territory.

    And I have, already in this brief career, been everything from a constructive and supportive reviewer to a lazy and (I assume, from the authors' perspective) rather dickish one. Seems to me the first thing to ask someone who is convinced that peer review is broken is, "how many manuscripts are waiting on your anonymous comments, then?"

    ----------------
    *It was possible to demonstrate, from the text of the review, that the reviewer had not read the manuscript, or at least not ready it very closely.

    • Martini says:

      Had the same experience. Again, the editors I have worked with will readily dismiss a these types of reviews if you can specifically point out the flaws in the reviewers comments.

  • gerty-z says:

    I think that when you get a "idiot" review can be fairly random no matter how much we look for correlations. I kinda wonder how many folks actually think that peer review is broken in a "and I will do something to change it" kinda way. Sometimes I think that bitching about the third reviewer is just part of the culture but not something that we actually think is a problem.

  • dftchemist says:

    Godd insight- abt bitching being part of the culture. Like kids griping when mum or dad comes by to take a look at their homework and making them do their corrections. It sucks, but sure does pay off!

  • DrLizzyMoore says:

    Recently, I have seen some pretty vicious reviews. I think that some of this is just the climate right now. Grants are getting rejected left and right. Papers are hitting the editorial shredder. But in all fairness, I do know labs that pull stuff together and throw it out for peer review for the reviewers and editors to sort out. It seems like scientists in general are on edge and the reaction is not healthy.

    • gerty-z says:

      do you think the vicious reviews are more frequent (or could it just be random?). And if they are more frequent, is it worse for the n00bs like us?

      • DrLizzyMoore says:

        I do think that vicious reviews are more frequent. I don't know if it is worse for us n00bs, or if we are more likely to take it personally than say a tenured person that's been at it for 20 years and maybe seen worse. Does that make sense? In full disclosure, i still have not submitted my first paper from my new job (my 1 year anniversary is coming up in a couple of weeks)-so the reviews that I see are either the other reviews I see on papers that I have reviewed or from papers from labs past (in which I contributed to the project). It's almost like *everyone* has this 'get off my lawn!" mentality.

        Like you, I try to be equitable and fair with reviews. I can only control myself and hopefully not contribute to the overall vitriol.

  • Spiny Norman says:

    I think there's truth in what Lizzy says.

    I've gotten a couple of genuinely nasty reviews since I became an Asst. Prof., and so has the asst. prof. who was hired immediately after me ("I recommend that all future manuscripts submitted by this group should be subjected to special scrutiny") was the last line of one of those. That's a nearly verbatim quote. It might have been worse.

    A major problem IMO is the now-pervasive inclusion of supplementary information, a seemingly good idea that turns out to be a colossally bad one. J. Neurosci. and J. Exp. Med. are leading the way by largely jettisoning supplementary materials sections. I think that this is a huge step forward.

    • gerty-z says:

      ouch! that is nasty. Do you think that reviewers are harder on asst. profs (and if so, why)?

      Also, I totally agree with you re: Supplementary materials. I wish all journals would get rid of them.

      • Martini says:

        Wow, that is bad.

        I agree, supplemental data needs to go. 5 figs with good data is all that should be needed.

      • Spiny Norman says:

        I used to think that reviewers were harder on junior faculty, and they may be. Times are hard and editors's expectations are very high, sometimes preposterously so. You're a biochemist? Give me a thorough analysis of an animal phenotype (preferably a mouse). You're a physiologist? Give me a biochemical mechanism.

        I definitely think it's harder to get your work sent out for review at IF >12 journals until you become more visible in a field.

        But the biggest factors are learning how to deal with editors, how to write a cover letter. You're got to learn how to be persistent and bounce back rapidly from a rejection with a strong resubmission, and how to coldly evaluate whether negative comments in review are mere stochastic dipshittery (everyone, but everyone, gets this) or valid criticisms that really need to be addressed prior to resubmission.

  • Pascale says:

    I have seen a paper reviewed where one reviewer basically requested that all experiments be repeated in a slightly different manner. No hyperbole, and it led to a rejection. The authors contacted the editor to complain and ultimately got it accepted, but it took a lot of effort to overturn an unreasonable request that generated a recommendation of rejection.

  • drugmonkey says:

    J Neuro is not "leading the way", SN. All the journals that never put up with Supplementary Bullshittery in the first place were leading the way all along!

  • [...] Sept: OMG academic peer review is totally fucked and it never works and the third reviewer is a stupid idiot asshole!!!11!!!1! [...]

  • Feeling bitchy about peer review (again!) today, and decided to vent in your comments. 🙂

    Speaking for myself, my complaints about peer review are not just about papers being rejected - I get just as annoyed by papers that are accepted but which have to be edited in stupid ways to adhere to peer feedback. In fact, in many ways these are *more* annoying because the changes made will be permanently associated with my work, even if they are entirely asinine. At least with a rejection I have a chance of pulling better peers in the next lottery! 🙂

    My basic problem with the way we currently do peer review is that no-one is a peer to anyone else in any academic field now. The volume of publications, research lines, inter-disciplinary approaches etc. is so vast that it is impossible to select appropriate peers. All peers that review any submission are at best operating in a similar academic discipline, they are not 'peers' in the sense of having equivalent status, skills and experience, nor could they be.

    "In the end, the only way that peer review will get better is if we 'peers' decide to change it."

    I don't quite agree - I want to change it, I'm certain many others who conduct and receive peer reviews would like to change it, but how could we do so? The problem isn't a lack of desire for a new approach to peer review, it's the lack of an alternative system that could replace it.

    Could we get a peer review of the peer review process, perhaps? 😉

    Take care,

    Chris.

  • […] First, remove the stack of unread manuscripts from thine own inbox. Is peer review broken, or are we all just lousy peers? […]

  • Frustrated and Saddened says:

    Peer review is definitely broken. Peer review is designed ideally, to do one thing, and one thing only. Judge whether the science that was done, was done correctly, and contained all the reasonable controls, and that the results support the conclusions. Note that this last part does not require there be only one way of interpreting the results, only that the interpretation favoured can be reasonably defended. Who here with experience in the peer review process believes that peer review today even exists in the same universe of what I just described?

    The entire process in the case of publishing has been subjugated to the profit motive of the tabloid journals (almost all considered "high impact") more interested in sexiness than substance. Most begin by asking you to subjectively judge the quality of the science. Any scientist worthy of the title should gag at the very thought of this. Any of you who would argue you are fit to make such subjective assessments need to attend anti-arrogance training every day for the next five years to figuratively bludgeon such self-important political hackery from your supposedly clinically objective brains. Or you could change careers and become odiously rigid bureaucrats and put your real skills to use, thus freeing up space for actual scientific minds to do research. One could create the mutant love-child of Einstein, Feynman, Pauli, and many dozens of other laureates in a broad range of fields, and you still wouldn't produce someone qualified or capable of accurately judging the "quality" or "importance" of any given piece of scientific work. If I were wrong about this, long-term citation rates would at least correlate with the impact factor of the journal the paper is published in, and they do not even reach this most basic hurdle.

    As for peer review in the granting process, it would seem the problem is even worse. However, this could be fixed simply, by removing the inane and intellectually bankrupt policy of funding projects rather than scientists. Tenure, like peer review, exists ultimately for one purpose only. To eliminate any and all influence, political or otherwise, on the independent researcher with regard to his/her choice of research topic. Under the current environment, one must first convince your peers that you are doing "top" research...whatever that means. Then you must convince them that despite being a tenured associate, full, or even a distinguished professor of long standing, that you are capable of doing what you propose. Then you must convince them that the way you want to do the research (even if reasonable) is the "best" way to do it. The result of course is that it is absolutely ludicrous to suggest that any justification for the existence of tenure remains under the current regime. You will not only have to tailor your research to a given audience, you'll have to propose to do the research the way they would, or it will be triaged from the pile. Not for any substantive flaw, but because there isn't enough funding and any arbitrary criteria that can be used to winnow an impossibly large square peg of applicants to help it fit in an increasingly small round hole of funding will be enthusiasticly seized upon by the poor schmoes who have to review the applicants. And not to worry, because any in a position to disagree with such behaviour are in the same boat and will no doubt agree with the triage process, achieving a near instaneous agreement in ad hoc "peer review" of the granting criteria that any scientist wishing to publish or gain funding could only dream of.

    I've personally witnessed a review from PNAS where the arguments were so transparently ridiculous I legitimately thought the person might be suffering from a degenerative neurological syndrome. This is not hyperbole, I actually wondered how such a thing could be missed by the editor and was concerned by how obtuse one would have to be to miss it. (This was of course a "prestigious" scientist as editor.) I would add that the arguments were also all founded on a spectacularly flawed paper where the researchers contradicted their own conclusions from one paragraph to another. This in a journal so obscure, that after a week of asking various colleagues if their institutions had access to the journal, eventually a colleague at the NIH responded in the affirmative. How to approach this? My colleagues decided to approach it clinically, by referencing the flawed foundations of the arguments directly referring to the reference on which the reviewer relied. The response from the editor? "The reviewer is a prestigious scientist, and I trust his opinion." Forgive me, but I wasn't aware people were actually so brazen as to openly admit that "prestige" was a factor in peer review, let alone "opinion". This is not the only time or a rarity by any means. Another colleague on the same floor had a reviewer of his paper state: "there is nothing novel here, everything shown could be assumed from previous published results". Again, I wasn't aware assumption was the new scientific criteria for publishing. Would any of you like to submit your work for my review so I can tell you what assumptions you didn't have to prove and how you wasted your time doing the actual science?

    Don't pooh pooh these as isolated instances. They are not isolated, and we are in both cases talking about "high impact" journals. I have personally experienced many similar reviews. These are a few examples from one floor of one department I've worked in. I can list similar instances in previous departments where I've been employed, and countless similar examples given to me by colleagues at conferences. It is appalling that scientific discourse and peer review has been corrupted to such a degree, and what is worse, this inevitably leads to squandered resources and research dollars that are already far too scarce.

    One last point for you all to ponder and respond to. I recently read a report that some scientists were pleasantly shocked to hear some tea party politicians endorse the idea of doubling the NIH budget. Ignoring whether this is likely to actually occur or not, do any of you believe this would lead to even a reasonably modest 25% increase in the number of tenure track positions available? Personally, I think we would be lucky to see such a funding bump result in even a 5% increase in the number of positions available on the tenure track. The rest will be gobbled up by making the labs that already have a piece of the pie richer. "Peer review" will do what it has been doing for the last 3 plus decades. It will reinforce those in positions of influence at the expense of everyone else. This will further corrupt scientific inquiry and the peer review process with intellectually unjustifiable layers of subjectivity, bureaucracy, politics, and other forms of self interest.

    • gerty-z says:

      None of your complaints seem aimed at the "structure" of peer review. I don't think funding "the scientist" instead of the project would change anything that you raise as issues. It could make things worse - the only "people" funded would be the ones that are already well-known and, perhaps, those that trained with them. So what would be the better option? As to your last point, I'm not sure that increasing the # of tt positions is, or should be, the goal. There are plenty of underfunded labs right now.

Leave a Reply