Archive for the 'grants' category

grantsmanship and peer review

Jun 22 2017 Published by under academia, grants, jr faculty, on the job training

There was a conversation on twitter last night that got me kind of worked up. Potnia already posted about this - and there are some really good points about how study section works there. You should read it! But I have opinions, too, and maybe my perspective is a little different. It's very possible I'm putting on the meat pants with this post, but so be it. I'm basing this on my (limited) experience writing grants (I have funded R01) and sitting on study sections (once as ECR, several other times ad-hoc). And I'm only talking NIH here - I have no real experience with NSF or other government agencies.

The twitter conversation was, more-or-less, focused on the frustration of submitting a revised application to NIH (the A1) and getting a worse score than the original submission (the A0), even though you addressed the concerns raised by the original review. This led to Dr. Becca suggesting that there should be a mechanism that kept scores from getting worse. I love Dr. Becca. She is smart and genuinely good people and makes delicious cocktails. I know the frustration of having an A1 get a worse score than the A0. But a rule that A1 always have a better score than the A0 is just not a reasonable solution.

First, there is the real possibility that the A1 is just not as good as the A0. Reviewers have to evaluate the grant that they are reading. Maybe the revisions actually lowered the impact of the grant. It could be that the new experiments (or preliminary data or whatever) were just not compelling. They could "unfocus" the grant, make the central hypothesis harder to understand, etc etc etc.

Second, the A1 may actually be better than the A0, but it still gets a worse score. HOW?? The two things that pop into my head are:
A. There were a lot of other grants in that study section that were just really good. Each round, reviewers are supposed to score grants compared to the other grants they get for that meeting. So even if the A1 is "better" it may not measure up when compared to the other grants the reviewers are reading.
B. Different reviewers scored the grant this time. Even though NIH has standing (or sitting?) members of the study section, there is no guarantee that the same folks read your grant each time. Different people will have different perspectives, background, etc. They will have different views of what are "strengths" and "weaknesses". They will have different approaches to weighing the strengths and weaknesses. It's n0t awesome, but it happens. IME (limited though it is), the folks that reviewed prior versions speak up in study section even if they weren't reviewers this time around.

BUT! If you did what the reviewers wanted, they shouldn't get to give you a worse score!! I disagree (see above). But also, and this is really important, the reviewers don't tell you what to do. They only point out what they consider to be strengths and weaknesses of the application. If the SRO is doing her job, they will make sure that "what they should do" types of statements aren't part of the discussion at study section. Just because a reviewer notes something they consider a "strength" or "weakness" doesn't necessarily mean that it is something that was a major influence of the overall impact score. You can get a sense of what the score "drivers" were by reading the summary statements. Not the bullet points, but the paragraphs written by the chair of the study section (which should reflect the discussion about your grant) and/or the summary paragraphs written by each reviewer to justify the impact score they assigned (especially if your grant was not discussed).

It is sometimes not straightforward to read the summary statements. Because reviewers are limited in what (and how) they can raise concerns. In a perfect world, your PO (NOT the SRO) would help you parse these statements. Or you will get help from an experienced mentor. Because if you misread the summary statements, you may think you are making the grant better by doing what the reviewer wants -- and then you inadvertently make the grant less exciting to that reviewer. It sucks. I joked on twitter that reading summary statements is like trying to translate an ancient prophesy. It's kind of true. But only because you sort of have to know where the reviewers are coming from and be able to speak their language. You only can learn this by practice. I sat on study section as an ECR (Early Career Reviewer) - it was more helpful than I can explain. I learned how other folks read grants. I experienced how discussions go, and how that translates into scores. And I have been lucky that I have great colleagues that let me read their reviews and that have read mine and given me advice. I'm lucky (so fucking lucky) that I have these resources. I searched them out, because a lot of really smart people (like DrugMonkey) made it clear to me it was important.

Funding lines are low. And when they are low, you need a great score to get funded. And you have to get lucky to get a great score. The reviewers need to align more precisely than the stars. So you need to do everything you can to get the reviewers on your side. That, in my mind, what "grantsmanship" is. It's sort of like learning a new language and the customs/rituals of a weird culture. But all you can do is try to make it as easy as possible for the reviewers to be on your side. And then you have to get lucky, too.

 

No responses yet

Of course there is bias in peer review, jackass!

Jun 08 2014 Published by under grants, venting

So the NIH has decided they want to try to minimize the effect of bias in peer review. That doesn't sound like a shitty idea, right? No. But apparently a lot of jag-offs couldn't stop themselves from starting in on the comments section. Many were "insulted" that anyone would accuse them of bias. Or wanted to make sure everyone knew that the only real bias was the kind that ended up tanking their proposals. Sigh. I don't even know how this could be a thing. Peer review is done by humans. Human are biased because of their previous experiences and their interactions with the culture and society in which they live. You don't have to be a fucking psychologist/sociologist to know this. You just have to have a few working neurons that can fire coherently and generate thoughts. Seriously, if this is hard for you then you DEFINITELY need to head over to DrugMonkey's blog to read the guest post by MyTChondria (who has a shit-ton of very active and coherent neurons, ftr).

I just have a couple things to add to some of the commenters over at Rock Talk. Beyond the obvious "pull your heads our of your assess, people!", of course that is addressed so well by MyTChondria.
1. If reviewers are repeatedly making "factual errors" when they review your grant, you might want to reconsider how you are writing your grant. Sure, reviewers will make mistakes (they are human, remember?). But it is a hell of a lot easier for a reviewer to be confused if you writing is jumbled, rambly, or incoherent.
2. There are not enough people with more than 3 R01 that limiting the number of awards would make a big difference. And if someone is smart enough and has enough ideas and resources to manage more than 3 grants, then why would we discourage them? There are a lot of places where it takes 2 R01 equivalents to run a minimal lab (soft money positions in particular). I know DrugMonkey has gotten into this before (for example, here).
3. Full-time reviewers. Seriously?!?!??! Who would take this job? No doubt they would never make mistakes. I assume that no one will be complaining about these "professional reviewers", same as everyone loves the non-academic journal editors. Can't have it both way, folks.

I just can't even understand how so many "scientists" can get their collective undergarments so twisted up over this topic. Come on folks. We're better than this.

12 responses so far

Better than a kick in the teeth!

Oct 05 2012 Published by under academia, grants

I woke up this morning crazy stiff from my workout with the new trainer yesterday. And then I just spent a good portion of my day in the dentist office, unexpectedly. It totally, totally sucked. But then the shitty day took a turn for the better when I got some happy grant news! My R01 that went in in June was in study section yesterday - and MY. GRANT. WAS. SCORED!!!! I had been hopeful when I hadn't gotten a "not scored" email last night, and even more guardedly optimistic when it was not waiting for me this morning. But I have now checked into Commons and there is real evidence (you know, a score!). That's right, no triage this time baby! I got a real score! Now, it is probably* not a fundable score, but it is an improvement. And scored means I get a summary statement and then I can resubmit. WOOOT!!!1!!11!!!1!!!!!

 

*but really, who knows. There is no budget yet, so that doesn't help. But I'll have to wait to see what my PO says about it after I get the summary statement back. Never can tell and all 🙂

6 responses so far

Is it cheating to propose experiments in a grant that are "done"?

Sep 27 2012 Published by under academia, grants

If you have already FINISHED the studies, why would you propose them in a grant? I know that there is a common meme that "you have to have done the work to get it funded", but I have never behaved that this is an actual real expectation.

This comes up because Fred, a new PI, left a comment over at Dr. Becca's place*:

...we are seriously thinking about publishing most of the preliminary data (and data not shown in the grant) about 1-2 months after submitting the grant. This manuscript would show we have completed more than half of proposed studies in the grant.

Fred wanted to know whether it is a good idea to publish the paper. But I am wondering WHY that was the grant that was written. If you had done the experiment wouldn't it be cooler to talk about what new awesome thing you could do next? I get that you need preliminary data for a grant. I have always written my grants with preliminary data that showed I was able to do the experimental procedure. I have always written my grants using Preliminary Data to demonstrate feasibility. But the reason I needed money was to actually DO the experiment. I may "know" the answer (or at least think that I do), but I haven't actually done the experiment.

It feel like holding back data, and proposing experiments that you have already done is kind of sketchy. And I don't understand the motivation. Is the idea that you will be able to show fast progress on the grant (if it's funded) and that will be awesome?

Help me out, guys. Am I totally off base here? What am I missing?

Also, FTR, in my n=1 experience on study section, if there is evidence that you have already published the results of the proposed research it is viewed poorly by the reviewers.

 

*I started to comment over there but realized that I was going off on a tangent.

30 responses so far

Gerty goes to study section, part 2

Jun 14 2012 Published by under academia, grants, on the job training

In this installment, I'm going to talk about what I did BEFORE I even got to study section. Obviously, everything I say (ever) is specific to my own experience. YMMV, etc. If you want to read about how I ended up as an ECR, see part 1.

After I was invited to study section, the real work started. I had to actually review grants! As an ECR (what a convenient acronym!), I had a reduced load. I only had to review about 30% as many grants as the "real" members of study section. At NIH, grants all get reviewed by 3-4 different people, who are cleverly named the "primary", "secondary", "third" and "fourth" reviewer*.  All reviewers that were new to this study section had an "official" training conference call with the SRO to make sure we were all on the same page. We talked about the focus of the study section, the scoring protocol, different grant mechanisms, and how to deal with ESI/NI applications (there was no difference in these two categories, as far as I could tell). We also all filled out the conflict-of-interest form. The NIH had identified folks you may have a conflict with (you work at same institution, have recently collaborated, etc), but you are expected to note any others that may be perceived as a COI.

Then, the scoring. NIH grants are scored for five criteria: Significance, Approach, Innovation, Investigator, and Environment. You can tell they are important because they are always written as proper nouns. Each criteria gets scored on a scale of 1-9 (PDF of scale),  1 is "exceptional", 9 is "poor". And then you list bullet points of strengths and weaknesses - that should justify the score you gave, of course! If you list a lot of weaknesses it will be hard to defend a "3", for example. In order to prevent score compression, you are encouraged to consider a 5 as average. We were all told that if we score something better or worse than 5 we should be able to justify why. In addition to the criterion scores, each grant gets an overall Impact score. The impact score is the Big Deal. There is no formula that derives the Impact score from the criterion scores. Instead, you have to decide what are the most important aspects of the application (both plus and minus). Then you decide what the potential for the proposed research to have a long-lasting and significant effect on the field is. This is the Impact score. Different factors contribute to the Impact score for each application. Some get lifted by the Approach, others by the Significance, and some by Investigator. At least one reviewer writes a paragraph explaining the Impact score that they assigned, so that the grant writer knows what was driving the score.

The point of the ECR program, at least in part, is to groom us young-uns to be good reviewers. Therefore, even though I was third reviewer for all of my grants, I did a full write up for all of them, including paragraph justifying my Impact score. I was expected to have mine done well before everyone else, so that the SRO could look at them. I was NOT given any direction on what to say, but ze looked at my reviews to give me feedback on whether the comments I made were consistent with the scores I was giving, and if I was going into enough/too much detail in each section. I actually found this really, really helpful. I have no personal experience with the new scoring system. The last NIH grant I wrote (my K99/R00) was still from the 25-page format and old scoring procedure.

It was surprisingly hard to give out bad scores! But I do what I gotta do. For each of my grants, I did some literature searches and read up so that I had an understanding of the field. Obviously, all of the grants were in my Field, but some were outside of my own sub-field expertise. This was important to help me figure out what the "impact" of the research might be [HINT: spell out what the "impact" will be when you are writing a grant!!!!] In addition to "my" grants, I read the Aims page of all the grants we would be reviewing (THIS IS WHY YOUR AIMS PAGE IS SO IMPORTANT). Then there were a couple other grants that I read because they sounded interesting, were by folks I knew, or were somewhat related to what I did.

A week or so before the meeting, final reviews are due. After all the reviews are in, you can see what the other reviewers thought of all the grants (except those you have a conflict with). I was Freaking EcstaticTM when my scores were - more or less - in line with what others had to say about the grants. It was also interesting to see what others picked up on, or missed. And it was REALLY interesting how different factors were weighted to come up with the Impact score for each grant. Sometimes, the Approach was the most important thing, other times the Investigator score  was really important, etc. But...these are the preliminary scores! Everything can change at the meeting.... stay tuned for part 3.

(How am I doing with the cliff-hanger here?)

 

*Don't ask me, I don't make the rules. Maybe, "tertiary" and "quarternary" are hard to say.

13 responses so far

Gerty goes to study section, part 1

Jun 12 2012 Published by under academia, grants, on the job training

Things have been pretty crazy, tweeps. I have some good news and some bad news.

First, the bad news: my first grant got triaged 🙁 I am now waiting to get the reviewer comments, and then I will talk to my PO and evaluate whether it is worth resubmitting as an A1 (chance of going from triage -> funded very, very low). Alternatively, I might re-work this application so much that it will go back as a new A0. We shall see.

Second, the good news: I didn't get fired! Woo Hoo 🙂 I got an official letter from the dean saying that I have been reappointed for the second three-year term. At the end of which I must be promoted to Associate with tenure or...not.

But in the last week I have also had the chance to sit on a real-live NIH study section. And I learned SO MUCH. And I want to share my new-found knowledge with YOU, of course. PLS had a great post from when he was on an NSF review panel, so hopefully my experience on this NIH study section will also be useful. Therefore, consider the rest of this post the first installment of the new series "Gerty goes to study section".

I sat on study section as an Early Career Reviewer (ECR*). This is a relatively new program to get young-uns involved in peer review so we will be better peer-reviewers and better grant writers. You can apply to be an ECR (there is an email link at the page linked above), but that is not how I stumbled into this gig. I actually ended up chatting with the SRO of a study section after giving a talk at a society meeting a couple of years ago. We were talking about the new ECR program, and I mentioned that it was something I would really like to volunteer for. Then a few months ago, I got an email from SRO asking if I would join in! Of course I said yes. And that, friends, is how I ended up on study section.

 

*because, right, that needs it's own acronym. Sigh.

18 responses so far

finding small pots of cash

Feb 08 2012 Published by under academia, grants

The vast majority of my time and energy as an assistant professor is spent trying to get funding for my lab. Without money, there is no one to do my kick-ass science. Start-up funds don't last forever, and (at least at my MRU) if you don't get an R01 from the NIH you won't get tenure. Even if you don't need an R01 in your gig, if research is a big part of your job you are probably expected to secure some sort of federal funding. So, yeah. You are gonna write applications for the big grants.

But at many places there are other, smaller pots of money that you should also keep an eye out for. These can be called "pilot grants" or "seed money" or "intramural funds". These are usually small grants, IME from 10-50K/yr. They generally last only 1 or 2 years. Just enough to do a fun experiment, develop a reagent, or pay part of a salary. In the first years my lab has cobbled together a not-insignificant amount of cash from the pilot/seed programs around here.

I <3 THE SEED $$!!

The question is: how do you find the seed money and get some for yourself? This is clearly going to be specific to your home institute/MRU. If there are any consortium or project grants around these sometimes will have funds for pilot grants. Some Uni will also use some of the money they earn from licensing IP to fund new pilot grants. You may have to keep your ear to the ground, because these opportunities tend to pop up without much warning. Another great thing is that the money can also show up pretty quickly. I have had less than 3 months from application to budget number, for example.

The applications are generally short. You need to propose a project that you can do in just a year or two, after all. It is not realistic to drop a 3-Aim R01 on a pilot grant. When I write for a pilot grant, I try to make it as explicit as possible how doing the proposed research will set me up to write an R01 (or equivalent) in the future. Most of the seed money sources that I am familiar with really want to know that they did something to start up a new project and that their money has been leveraged into something bigger.

The review of these grants can be internal, or your grant could be sent out to external reviewers. Either way, you will want to be familiar to the folks that have this kind of money and run these programs. You gotta get to know the folks that are running the big project grants in your area. Just like any other area, networking is a Good Thing. Even if there isn't a pilot grant on the line (now), you want these BSD folks to know who you are. Pitch your research program to them, and see what they think is most interesting. I have used these kinds of interactions to get an idea of how folks outside my MRU are gonna respond to different research ideas. And it is helpful to know what outside folks will think are the weaknesses, so you are ready to defend them.

There is almost nothing to lose from applying for pilot grant seed money. IME, junior faculty can be really successful in getting these kinds of funds. In fact, some of these programs are actually LIMITED to us jr. faculty. And it may be that more established researchers aren't going to go through the trouble for such a little pile of cash. But little piles of cash can be a big deal when you are starting out. You can generate some preliminary data, and get some feedback on a future Aim for a Big Grant. So, fellow assistant professors: I say, go for the seed money! Good luck 🙂

7 responses so far

Ripped from the headlines!!

Nov 30 2011 Published by under awesomeness, grants, queer

I am in the midst of writing an R01, and I am struggling. Because this grant really, apparently, wants to be TWO grants. But I don't think I should submit two applications this go-around. I'm afraid that I won't do as good of a job as if I focus all my energy on just one. I have spend the last hour or two typing and erasing to try to cut through what is floating around in my head. In doing this, I am basically writing rough drafts of the Aims pages for both. Then I will run them by folks to see which seems to resonate the best. All I can say is that I am very lucky to have many folks that are willing to let me bounce ideas off of them. And that is awesome.

But, of course, none of that is "Ripped from the headlines". No. But a little bit ago I got a little distracted from my grant and started clicking around on the internet. And I ran across a couple of things that I want to share (in the hopes this will clear out a little of the clutter in my head right now).

1. Have I mentioned that Emily Hauser is pretty kick ass? I thought so. In case you haven't started reading her blog, go over there now. She has a super breakdown of how treating adultery the same as sexual harrasment/assault contribute to rape culture. Excellent stuff, as usual.

2. If you have a friend that is LGBT/Q, you have already seen these two videos. But they are AWESOME. If you wanna feel good about people, watch them.

Zach Walls, son of a lesbian couple, speaks to the OH IOWA legislature

The best ad ever for marriage equality

4 responses so far

What a great Tuesday!!

Jun 14 2011 Published by under awesomeness, grants

I have just got news that one of my pending grants will be funded. It is not the holy grail of the R01, but it is a big deal for me. YAY!!

This news was greatly appreciated, especially after the last few days that i have spent in a "workshop" for my subfield. There have been some good interactions, but for some reason there seems to be a trend of senior folks being rather condescending. It is a little exhausting.

One thing that I have noticed over the course of this workshop, however, is the difference in the way that young women and men present their research. Each of the trainees (senior postdocs and new junior faculty) at this workshop presents a potential grant outline in order to get pre-submission feedback. Many of the data are preliminary, as they should be. What i have noticed is that most of the females are timid, almost apologetic, during these talks and the question and answer session. The men don't seem to have this problem, and do a much better job of defending their proposals. I don't understand this at all. The women here are successful and have good data and ideas. So why are they so less confident? I really don't understand this phenomenon at all.

Ladies, we have to do better!! I get that it can be intimidating to stand up in front of a small group of the big names in the field...but that is no excuse. Do whatever you have to in order to appear more confident. FAKE it if you have to. Try to emulate the behavior of good (confident) speakers, male and female. Practice more so that you are more comfortable. But we have got to stop standing up and seeming so timid. If you don't (seem) to believe in your data, there is no way your audience will.

12 responses so far

Grants v. Papers

May 21 2011 Published by under grants, jr faculty, on the job training

Well, it is at least 5 h after the rapture should have made it to my time zone, and I'm still here. Not a surprise, really. I was never really that worried about whether I would be raptured. One of the benefits of being an atheist, I suppose.

One thing I have been thinking about a lot recently, though, is the writing of papers versus grants. Specifically, given that I have finite time resources, which should I be focused on right now?

Clearly, I need to have both grants and pubs to get tenure. I am approaching the first anniversary of my faculty position, and I have submitted many grants with mediocre success. I have one paper in review right now, and another that will go out soon, but these both still have my postdoc PI as an author, so even though i am the corresponding author they only kinda count.

I have decided that this summer I will get back to the bench and focus on getting some data, moving projects forward and getting at least 2 manuscripts out the door. I feel like right now I have to focus on publishing. But I really don't know if this is the best strategy. Sure, eventually we will need to show productivity, but it is conceivable that I could ride the momentum that I have now and get more cash.

I really don't know the correct answer here. But I'm s little burned out on grant writing. And I would like to see something actually happen. I also think that me being at the bench more this summer could be good to get my two grad students started off on the right foot.

Time will tell.

28 responses so far

Older posts »