Of course there is bias in peer review, jackass!

Jun 08 2014 Published by under grants, venting

So the NIH has decided they want to try to minimize the effect of bias in peer review. That doesn't sound like a shitty idea, right? No. But apparently a lot of jag-offs couldn't stop themselves from starting in on the comments section. Many were "insulted" that anyone would accuse them of bias. Or wanted to make sure everyone knew that the only real bias was the kind that ended up tanking their proposals. Sigh. I don't even know how this could be a thing. Peer review is done by humans. Human are biased because of their previous experiences and their interactions with the culture and society in which they live. You don't have to be a fucking psychologist/sociologist to know this. You just have to have a few working neurons that can fire coherently and generate thoughts. Seriously, if this is hard for you then you DEFINITELY need to head over to DrugMonkey's blog to read the guest post by MyTChondria (who has a shit-ton of very active and coherent neurons, ftr).

I just have a couple things to add to some of the commenters over at Rock Talk. Beyond the obvious "pull your heads our of your assess, people!", of course that is addressed so well by MyTChondria.
1. If reviewers are repeatedly making "factual errors" when they review your grant, you might want to reconsider how you are writing your grant. Sure, reviewers will make mistakes (they are human, remember?). But it is a hell of a lot easier for a reviewer to be confused if you writing is jumbled, rambly, or incoherent.
2. There are not enough people with more than 3 R01 that limiting the number of awards would make a big difference. And if someone is smart enough and has enough ideas and resources to manage more than 3 grants, then why would we discourage them? There are a lot of places where it takes 2 R01 equivalents to run a minimal lab (soft money positions in particular). I know DrugMonkey has gotten into this before (for example, here).
3. Full-time reviewers. Seriously?!?!??! Who would take this job? No doubt they would never make mistakes. I assume that no one will be complaining about these "professional reviewers", same as everyone loves the non-academic journal editors. Can't have it both way, folks.

I just can't even understand how so many "scientists" can get their collective undergarments so twisted up over this topic. Come on folks. We're better than this.

12 responses so far

  • DrugMonkey says:

    But, but...... Their feelings were hurt, badly, by the implication they might have biases!

  • namnezia says:

    And they've been on "hundreds" of study sections and never seen evidence of this!

  • iGrrrl says:

    About this professional reviewer thing, and how they should be able to go to meetings to keep up with the scientific progress?

    Ears, comma, steam, comma, out of.

    I am, essentially, a professional reviewer. I've been away from the bench since 2001, and I can barely keep up with the technical advances, much less the literature. I look for presentation of the argument, errors in logic, ways to improve communication (see Gerty-z's first point, above). My clients are strongly urged (required) to solicit peer review for scientific and technical merit, because I can't do that for them in most cases.

    Over beers some weeks ago, DrugMonkey noted that when called to review a proposal outside one's field, as happens on study section, review groups for F and K awards, etc., the reviewer essentially finds themselves reviewing the document. In other words, reviewing it for the things that I teach, not necessarily the science. I've been thinking about that comment ever since.

    The reason steam comes out of my ears at the idea of professional reviewers is that it would probably reduce the grant game to a document game*. Sure, packaging and presentation of the idea (grantsmanship) helps to get your idea across, but only people current in the field are really capable of judging merit. My job is just to help people present that argument clearly, so peers can judge the science, because when it comes to the science, I am not, as professional reviewer, their peer.

    *Some would argue that it is already; I'm not so sure.

  • DrugMonkey says:

    Don't forget the other driver on the tendency to conduct a review of the document. It is an entirely laudable attempt to be *fair*.

    "If we're going to beat up this application for failure to clearly state a hypothesis then we have to beat up ALL applications that fail to do so"

    This, even a well justified exploratory effort gets nailed. For fairness sake.

  • Scientists aren't specially qualified as individuals to be "objective" or "unbiased", and it is delusional (and destructive) in the extreme to think that we are. It is exactly this delusion that leads to people like Hauser faking up loads of data, because they "know" their hypotheses are "correct". Scary how much of that shit shows up on any hot-button thread at Rockey's blogge.

  • DrugMonkey says:

    ...but scientists imagine that they are objective because of the nature of the job, PP. It's an occupational hazard!

    The problem comes in when they abandon their professional standards for evidence and evaluation.

  • Dave says:

    Professional grant reviewers sounds almost as bad as politicians without term limits

  • Mytchondra says:

    I think you're looking for the phrase "MyTChondrias CoBlog with DrugMonkey" .

    Sweet bejebus...how much Chinese take out did you eat before I showed up, Ted?!

Leave a Reply