Gerty goes to study section, part 3

Jul 15 2012 Published by under academia, tenure-track OTJT

It is about time! I got a little distracted, but it is time to finish telling you all about my study section experience. In case you can't remember, in part 1 I wrote about how I ended up as an ECR on an NIH study section, and in part 2 went on about what I did leading up to the meeting. Now part 3 - the meeting!

You guys, the meeting itself was exhausting! We started at 7:30 am, and went until 6:00 pm. We were meeting in a conference room with a giant table. Everyone had an assigned seat, and you got a seating chart so you would know who everyone is. The first few minutes are milling about and getting coffee/pastries*, setting up your computer and getting it connected to the wireless, etc. Then the meeting starts with the SRO welcoming everyone and explaining the rules. You all check your COI form to make sure that it is correct. Then she hands the meeting over to the Chairperson of the study section.

The Chairperson then starts the review process. The first grant is announced, and anyone that is in conflict leaves the room. For each review, it basically went the way it is shown in the movie made by NIH. Each reviewer announces their primary impact score. Then the first reviewer gives a short description of the grant and what were the strengths/weaknesses that led to its score. Then the other reviewers add anything that they considered to drive their score. At this time, everyone else is listening and looking over the grant, particularly the Project Summary and Aims but you could pull up the whole grant on your computer.

After all the reviewers spoke, then anyone could ask questions. There were lots of questions. Questions could be about the science or the scoring. For example, you could ask a reviewer why they gave a grant a 1 but had many concerns about the approach. Or you could ask for clarification about what experiments they were (or were not) proposing. Or you could ask about the interpretation of the preliminary data. The reviewers answered the questions (which is why it is so important to make sure they have all the information they need to advocate for your project!). After 10 minutes or so the Chairperson summarizes the discussion, and asks the Primary Reviewer if there are any concerns about animal use or reagent sharing (seriously people, don't forget to fill these sections out). Then the original three reviewers announce their final impact score, which may (or may not) have changed. This sets the "range". The Chairperson then asks if anyone will be voting outside the range, and you have to raise your hand if you are. If you didn't say anything at all during the discussion but then indicate you are voting outside of the range, the Chairperson may ask why. They mostly want to know if it was something in the discussion or something else. Then everyone enters their score on the online form. And then on to the next grant.

The order that the grants were reviewed was based on preliminary impact score. The R01 from ESI were the first group, with the "best" scored grants first. We only discussed the top half, the rest were "not discussed" (aka triaged)**. But every grant was announced and we all had to agree to triage, so if one person wanted to discuss the grant, then it was discussed. After the ESI R01, we went through all the top half of the rest of the R01 applications. Midway through this group we got a 30 min break for lunch (seriously, it was a LONG day). Then we started on the R21 and R03. There were so many R21. SO MANY. After about 10 hours of this, we stopped and all of us went to dinner, which was pretty awesome. The SRO made sure I got introduced to everyone, especially the BSD in my field that were there, which was awesome. The next morning we met again to finish the R21. We worked through lunch, and finished early in the afternoon. It was crazy.

The whole process was fascinating, really. We all know that there is no formula that derives an impact score, but it is really amazing how each grant was evaluated differently. For some grants, the approach was KEY. But there were some where the reviewers noted significant concerns about the approach but still gave it a very good impact score because of the investigator, or vice versa (for instance). Every so often the SRO would interrupt the discussion to make sure that we were following the rules. For instance, we were not allowed to say "This grant would be better if they had controlled for widget frequency". You could only raise concerns. This was to prevent people from assuming if they did everything the reviewers "suggest" that they would get a better score. Because who knows? There were also some times that the SRO would ask someone to clarify their score, if it wasn't clear that they were following the scoring system rubric. This was to try to avoid score compression. There was also a program officer there, but she didn't say anything during the meeting, just watched.

Being on study section was really interesting experience. I think that I knew many of the concepts of what was happening (thanks for folks like DrugMonkey and PhysioProf, etc.). But seeing it in person really brought it all home.

Now all I have to do is translate this experience into my own successful grant!

Do you guys have questions, or requests to expand on something? I don't have a plan for a Part 4, but I am open to suggestions.

 

*apparently this is the last meeting that will have these, though. New rules prevent the NIH from buying food or coffee for these meetings.

**You get notified that day if you are triaged, based on my personal experience. But you don't get to see your preliminary scores for a while after that.

8 responses so far

  • Dr Becca says:

    Awesome series, Gerty! Thanks so much for sharing all of this.

  • Hermitage says:

    Yesh, thank you for sharing all the sekrets! I shall inscribe it in a fine leather-bound book.

  • Bashir says:

    So it was pretty much by the book? No interesting nuggets other than how long the day is?

    Was pretty much everyone senior or were there other junior folks?

  • Isabel says:

    How many people were in the room with you?

    Were there any big arguments, any protracted (relatively speaking of course) battles for particular grants?

    Thanks for this series btw.

  • gerty-z says:

    The full roster of any study section can be found online (see here: http://www.csr.nih.gov/Roster_proto/sectionI.asp). There are usually ~25 people on each study section, and many of them are well-established, mid-career. You (generally) need to have an active NIH grant (usually an R01) to sit. I think it is harder to recruit junior faculty that have a grant but are not tenured yet. My dept. chair has made it pretty clear that doing study section for real would be a bad idea before I was promoted, even though they were supportive of me serving as an ECR.

  • gerty-z says:

    There were some big arguments, but I can't really go into details. Every discussion is supposed to remain confidential, first of all. Second, I would like to preserve my pseudonym. The really well-scored grants (which presumably will be the only ones funded) had relatively little discussion because everyone pretty much agreed they were outstanding. The longer the discussion about a grant, the higher (worse) it's score seemed to turn out, which was interesting. This was true even for grants that started out with impact scores of 1-2 - a long discussion could end with them getting 2-3, for example. But this was just a trend and I definitely saw some folks really fighting for a grant and it ended up moving the score down. It really is true if there is one passionate advocate for you on the study section it is a Good Thing. The most contentious discussions were often about A1's that were being scored, particularly if the PI didn't have other funding. This was true both for well-established and respected PIs (that may have to shut down their lab) and younger faculty that were just getting started (but maybe didn't get the ESI bump).

    Really fascinating, to me, was that there were some instances where established PI had included junior faculty as Co-PI. How this was perceived varied depending on circumstances, but at least some times it was viewed negatively if the senior PI was being seen as "taking advantage" to the junior person.

  • AD says:

    Thanks for this series. It made interesting reading.

  • GMP says:

    Interesting. This sounds almost exactly like an NSF panel.

Leave a Reply