Gerty goes to study section, part 2

Jun 14 2012 Published by under academia, grants, on the job training

In this installment, I'm going to talk about what I did BEFORE I even got to study section. Obviously, everything I say (ever) is specific to my own experience. YMMV, etc. If you want to read about how I ended up as an ECR, see part 1.

After I was invited to study section, the real work started. I had to actually review grants! As an ECR (what a convenient acronym!), I had a reduced load. I only had to review about 30% as many grants as the "real" members of study section. At NIH, grants all get reviewed by 3-4 different people, who are cleverly named the "primary", "secondary", "third" and "fourth" reviewer*.  All reviewers that were new to this study section had an "official" training conference call with the SRO to make sure we were all on the same page. We talked about the focus of the study section, the scoring protocol, different grant mechanisms, and how to deal with ESI/NI applications (there was no difference in these two categories, as far as I could tell). We also all filled out the conflict-of-interest form. The NIH had identified folks you may have a conflict with (you work at same institution, have recently collaborated, etc), but you are expected to note any others that may be perceived as a COI.

Then, the scoring. NIH grants are scored for five criteria: Significance, Approach, Innovation, Investigator, and Environment. You can tell they are important because they are always written as proper nouns. Each criteria gets scored on a scale of 1-9 (PDF of scale),  1 is "exceptional", 9 is "poor". And then you list bullet points of strengths and weaknesses - that should justify the score you gave, of course! If you list a lot of weaknesses it will be hard to defend a "3", for example. In order to prevent score compression, you are encouraged to consider a 5 as average. We were all told that if we score something better or worse than 5 we should be able to justify why. In addition to the criterion scores, each grant gets an overall Impact score. The impact score is the Big Deal. There is no formula that derives the Impact score from the criterion scores. Instead, you have to decide what are the most important aspects of the application (both plus and minus). Then you decide what the potential for the proposed research to have a long-lasting and significant effect on the field is. This is the Impact score. Different factors contribute to the Impact score for each application. Some get lifted by the Approach, others by the Significance, and some by Investigator. At least one reviewer writes a paragraph explaining the Impact score that they assigned, so that the grant writer knows what was driving the score.

The point of the ECR program, at least in part, is to groom us young-uns to be good reviewers. Therefore, even though I was third reviewer for all of my grants, I did a full write up for all of them, including paragraph justifying my Impact score. I was expected to have mine done well before everyone else, so that the SRO could look at them. I was NOT given any direction on what to say, but ze looked at my reviews to give me feedback on whether the comments I made were consistent with the scores I was giving, and if I was going into enough/too much detail in each section. I actually found this really, really helpful. I have no personal experience with the new scoring system. The last NIH grant I wrote (my K99/R00) was still from the 25-page format and old scoring procedure.

It was surprisingly hard to give out bad scores! But I do what I gotta do. For each of my grants, I did some literature searches and read up so that I had an understanding of the field. Obviously, all of the grants were in my Field, but some were outside of my own sub-field expertise. This was important to help me figure out what the "impact" of the research might be [HINT: spell out what the "impact" will be when you are writing a grant!!!!] In addition to "my" grants, I read the Aims page of all the grants we would be reviewing (THIS IS WHY YOUR AIMS PAGE IS SO IMPORTANT). Then there were a couple other grants that I read because they sounded interesting, were by folks I knew, or were somewhat related to what I did.

A week or so before the meeting, final reviews are due. After all the reviews are in, you can see what the other reviewers thought of all the grants (except those you have a conflict with). I was Freaking EcstaticTM when my scores were - more or less - in line with what others had to say about the grants. It was also interesting to see what others picked up on, or missed. And it was REALLY interesting how different factors were weighted to come up with the Impact score for each grant. Sometimes, the Approach was the most important thing, other times the Investigator score  was really important, etc. But...these are the preliminary scores! Everything can change at the meeting.... stay tuned for part 3.

(How am I doing with the cliff-hanger here?)

 

*Don't ask me, I don't make the rules. Maybe, "tertiary" and "quarternary" are hard to say.

13 responses so far

Gerty goes to study section, part 1

Jun 12 2012 Published by under academia, grants, on the job training

Things have been pretty crazy, tweeps. I have some good news and some bad news.

First, the bad news: my first grant got triaged 🙁 I am now waiting to get the reviewer comments, and then I will talk to my PO and evaluate whether it is worth resubmitting as an A1 (chance of going from triage -> funded very, very low). Alternatively, I might re-work this application so much that it will go back as a new A0. We shall see.

Second, the good news: I didn't get fired! Woo Hoo 🙂 I got an official letter from the dean saying that I have been reappointed for the second three-year term. At the end of which I must be promoted to Associate with tenure or...not.

But in the last week I have also had the chance to sit on a real-live NIH study section. And I learned SO MUCH. And I want to share my new-found knowledge with YOU, of course. PLS had a great post from when he was on an NSF review panel, so hopefully my experience on this NIH study section will also be useful. Therefore, consider the rest of this post the first installment of the new series "Gerty goes to study section".

I sat on study section as an Early Career Reviewer (ECR*). This is a relatively new program to get young-uns involved in peer review so we will be better peer-reviewers and better grant writers. You can apply to be an ECR (there is an email link at the page linked above), but that is not how I stumbled into this gig. I actually ended up chatting with the SRO of a study section after giving a talk at a society meeting a couple of years ago. We were talking about the new ECR program, and I mentioned that it was something I would really like to volunteer for. Then a few months ago, I got an email from SRO asking if I would join in! Of course I said yes. And that, friends, is how I ended up on study section.

 

*because, right, that needs it's own acronym. Sigh.

18 responses so far