In this installment, I'm going to talk about what I did BEFORE I even got to study section. Obviously, everything I say (ever) is specific to my own experience. YMMV, etc. If you want to read about how I ended up as an ECR, see part 1.
After I was invited to study section, the real work started. I had to actually review grants! As an ECR (what a convenient acronym!), I had a reduced load. I only had to review about 30% as many grants as the "real" members of study section. At NIH, grants all get reviewed by 3-4 different people, who are cleverly named the "primary", "secondary", "third" and "fourth" reviewer*. All reviewers that were new to this study section had an "official" training conference call with the SRO to make sure we were all on the same page. We talked about the focus of the study section, the scoring protocol, different grant mechanisms, and how to deal with ESI/NI applications (there was no difference in these two categories, as far as I could tell). We also all filled out the conflict-of-interest form. The NIH had identified folks you may have a conflict with (you work at same institution, have recently collaborated, etc), but you are expected to note any others that may be perceived as a COI.
Then, the scoring. NIH grants are scored for five criteria: Significance, Approach, Innovation, Investigator, and Environment. You can tell they are important because they are always written as proper nouns. Each criteria gets scored on a scale of 1-9 (PDF of scale), 1 is "exceptional", 9 is "poor". And then you list bullet points of strengths and weaknesses - that should justify the score you gave, of course! If you list a lot of weaknesses it will be hard to defend a "3", for example. In order to prevent score compression, you are encouraged to consider a 5 as average. We were all told that if we score something better or worse than 5 we should be able to justify why. In addition to the criterion scores, each grant gets an overall Impact score. The impact score is the Big Deal. There is no formula that derives the Impact score from the criterion scores. Instead, you have to decide what are the most important aspects of the application (both plus and minus). Then you decide what the potential for the proposed research to have a long-lasting and significant effect on the field is. This is the Impact score. Different factors contribute to the Impact score for each application. Some get lifted by the Approach, others by the Significance, and some by Investigator. At least one reviewer writes a paragraph explaining the Impact score that they assigned, so that the grant writer knows what was driving the score.
The point of the ECR program, at least in part, is to groom us young-uns to be good reviewers. Therefore, even though I was third reviewer for all of my grants, I did a full write up for all of them, including paragraph justifying my Impact score. I was expected to have mine done well before everyone else, so that the SRO could look at them. I was NOT given any direction on what to say, but ze looked at my reviews to give me feedback on whether the comments I made were consistent with the scores I was giving, and if I was going into enough/too much detail in each section. I actually found this really, really helpful. I have no personal experience with the new scoring system. The last NIH grant I wrote (my K99/R00) was still from the 25-page format and old scoring procedure.
It was surprisingly hard to give out bad scores! But I do what I gotta do. For each of my grants, I did some literature searches and read up so that I had an understanding of the field. Obviously, all of the grants were in my Field, but some were outside of my own sub-field expertise. This was important to help me figure out what the "impact" of the research might be [HINT: spell out what the "impact" will be when you are writing a grant!!!!] In addition to "my" grants, I read the Aims page of all the grants we would be reviewing (THIS IS WHY YOUR AIMS PAGE IS SO IMPORTANT). Then there were a couple other grants that I read because they sounded interesting, were by folks I knew, or were somewhat related to what I did.
A week or so before the meeting, final reviews are due. After all the reviews are in, you can see what the other reviewers thought of all the grants (except those you have a conflict with). I was Freaking EcstaticTM when my scores were - more or less - in line with what others had to say about the grants. It was also interesting to see what others picked up on, or missed. And it was REALLY interesting how different factors were weighted to come up with the Impact score for each grant. Sometimes, the Approach was the most important thing, other times the Investigator score was really important, etc. But...these are the preliminary scores! Everything can change at the meeting.... stay tuned for part 3.
(How am I doing with the cliff-hanger here?)
*Don't ask me, I don't make the rules. Maybe, "tertiary" and "quarternary" are hard to say.