Results of the Peer Review Capacity EvaluationCSR Advisory Council

5-18-2015

Luci Roberts, Director of Planning and EvaluationKatrina Pearson, Assistant Director, Division of Statistical Analysis and Reporting

Sally Amero, NIH Review Policy OfficerOffice of Extramural Research, NIH

2

RationaleUnrelenting Demands on Peer Review• Number of applications continues to increase• Number of grantees stays the same• 60% of NIH-funded investigators have served in NIH

peer review in the last ten years• 35-40% of NIH-funded investigators served in each of

FYs 2011 - 2014

Question: How can we continue to uphold the excellence of peer review while faced with increased numbers of applications and the same or fewer reviewers?

3

Approach• Question 1: What is a reasonable expectation for review service from NIH-funded investigators?

Surveys of SROs, reviewers and prospective reviewers to determine factors that contribute to: • SRO’s choices about whom to invite • Reviewers choices about whether to accept

4

Approach• Question 2: Which NIH–funded investigators serve as reviewers?

Using the IMPAC2 Person data, compare the characteristics of NIH-funded investigators who have served as reviewers to those who have never served, examine whether their characteristics have changed over time.

5

Question 1 What is a reasonable expectation for review service

from NIH-funded investigators?

Survey Inclusion Criteria

Applicant-reviewers: (n = ~4,000; 1830 total respondents; 46%)

– submitted at least one grant application as single PI or contact PI in the past five years (any activity code)

– had active funding from NIH in past five years (any activity code, including F, T, K, etc.)

SROs: (n = 423; 271 respondents; 64%)

- All SROs who had a SRG assigned to them during the two rounds prior to the survey

6

Respondent Counts• NIH Grantees – highest role on project (past 5 years)– Principal Investigators/Project Directors: 1,616– Subproject/Core/Consortium Leads: 37 (402 total)– Training Program Directors: 8 (145 total)

• Have worked on NIH Clinical Research in past 5 years(Defined in the Q as Research involving Human Subjects)– Yes: 754 (43%)– No: 1,011

7

No Yes0

10

20

30

40

50

60

70

80

90

100

55.32

43.9338.95

In the past 12 months, have you been asked to serve as a peer reviewer on a NIH Scientific Review Group?

Did you review for NIH?

Asked to Review

Reviewed for NIH

Perc

ent

8

During the initial phase of your effort to recruit reviewers,…

0

20

40

60

80

100

which categories/ranks of scientists do you consider inviting for review service?

Perc

ent

0

20

40

60

80

100

which kinds of prior experience on research grants do you prioritize?

9

Highest Project Roles of respondents asked to review (and not asked)

Invited Not invited0%

20%

40%

60%

80%

100%

Fellows

Training Program PI/Director

Subcontract/Consortium Lead

Subproject PI

Principal Investigator/Project Director

Perc

ent

Asked to Review Not Asked to Review

98%86%

10

Ranks of PI/PD respondents asked to review (and not asked)

Invited Not invited0%

20%

40%

60%

80%

100%

54%

22%

32%

23%

11%

37%

Assistant Professor or equivalent rank

Associate Professor or equivalent rank

Full Professor or equiva-lent rank

Perc

ent

Asked to Review Not Asked to Review

11

When you reviewed in the past year, was the review process more burdensome than it could be?

(and if yes, how so?)

No Yes Don't Know0

20

40

60

80

100

n = 630

n = 158n = 73

Perc

ent

Reasons given for “yes”: Number of applications assigned: 45%Too much time devoted to applications that will never be awarded: 16%Applications assigned to reviewers that do not have appropriate expertise: 5%Too many other demands on time: 5%Travel/meeting accommodations: 4%

12

0

20

40

60

80

100 Reasons to Accept (NIH Reviewers versus SROs)

Pe

rce

nt

Requirement to travel

from home

Time required to prepare the reviews

Competing responsibilities:

personal (family, social,

etc. )

Competing responsibilities: grantsmanship/research related

Competing responsibilities:

within own institution

(administration, teaching, etc )

Previous review service

(I've done my time)

0

20

40

60

80

100 Reasons to Decline (NIH Reviewers versus SROs)

Assistant Professor or equivalent rankAssociate Professor or equivalent rankProfessor or equivalent rankSeries4SRO (New NIH Reviewers)SRO (Experienced NIH Reviewers)

Perc

ent

13

0 1-2 3-4 5 or more

0 1-2 3-4 5 or more

0 1-2 3-4 5 or more

0 1-2 3-4 5 or more

0

20

40

60

80

100How many in-person review meetings per year

would you consider reasonable for a scientist like you?

ReviewerSROs

Perc

ent

1-3 4-6 7-9 10-12 13-16 1-3 4-6 7-9 10-12 13-16 1-3 4-6 7-9 10-12 13-16 1-3 4-6 7-9 10-12 13-160

20

40

60

80

100 How many applications per meeting would you consider reasonable for a scientist like you?

Perc

ent

R01/R01-like Fellowship/Career Multi-Project SBIR/STTR

14

0

20

40

60

80

100 Please estimate the percentage of professional effort you currently allocate to each of the following responsibilities

Assistant Professor or equivalent rank

Associate Professor or equivalent rank

Professor or equivalent rank

Pe

rce

nt

Less than 5% 5-10% 11-20% 21-30% Greater than 30%0

20

40

60

80

100 Ideally, what percentage of your professional level of effort should you spend on grant review service?

Perc

ent

15

What the surveys found….• Reviewers: time required to prepare reviews is most burdensome; most

widely selected reason to decline an invitation • Reviewers: Assistant Professors share similar motivations as more senior

investigators; grantsmanship experience is important at all levels• NIH’s service expectations exceed the reported capacity of most reviewers

to serve• More than half of the NIH respondents to the survey had not been invited

to review • Continuous submission and Flexible terms of service not a proximal

influence on the decision to accept an invitation to review• SROs have a good sense of what motivates reviewers to review:

– responsibility, – social networking, and – improved knowledge of grantsmanship and the science in their field

• SROs rely primarily on Principal Investigators, fewer SROs recruit from the other roles

Based on the survey results…

• Clear guidance on eligibility criteria for ad hoc service may be helpful – particularly criteria that do not narrow the pool of eligible reviewers as success rates shrink

• Better (searchable) access to information on investigators involved in other roles on funded grant may be helpful to SROs searching for reviewers

• Information about who has been invited to review would be really helpful for understanding how to improve NIH’s capacity for Peer Review

16

17

Characteristics of NIH Reviewers: Question 2

Which NIH–funded investigators serve as reviewers?

18

Data and MethodsConstructed a sampling frame from IMPAC II, FY

2009 -2014 containing all persons who submitted a competing grant application

Sub-analyses restricted reviewers to those with R01-equivalent or “Other” grant support excluding fellowship and career awards

ARRA Applications were excludedFunding amounts for multiple PI awards for

apportioned equally to each of the multiple PIs

R01-Equivalent Awardees = 25,294Other Awardees = 18,998

19

4,069 PIs total1,586 did not review

8,751 PIs total2,013 did not review

R01-Equivalent Awardees = 25,294Other Awardees = 18,998

20

5,741 PIs total2,101 did not review

8,340 PIs total1,576 did not review

In Conclusion – Investigators with the least amount of support served at a

lower rate

– Most investigators who have had R01-equivalent awards during the 5-year period have served as reviewers

– However, a significant number of PIs with > $500K in recent R01 support have not served as peer reviewers

– NIH Issued NOT-OD-15-035.html:

“The NIH expects principal investigators of NIH supported grants and contracts to serve on NIH peer review groups, when asked… Therefore, the NIH expects grantee institutions and R&D contract recipients to encourage their NIH-funded investigators to serve on NIH peer review and advisory groups. ”

21

Modeling Exercise• Assume that, in response to a large influx of new

applications, NIH needs to recruit more reviewers to participate in peer review.

• How many more reviewers could be enlisted from our own PI ranks?

• Assumptions: – Seek reviewers with at least one active award, at least

$500k in funding, and at least one R01 award in the past five years

– 80% of PIs who have not served agree to review for one meeting each

– 6 applications per reviewer/meeting on average22

23

1 to 2 3 to 4 5 to 7 7 to 10 Above 110

2,000

4,000

6,000

8,000

10,000

0

2,000

4,000

6,000

8,000

R01-Equivalent Awardees with Individual Funding Amounts of $500,001-$2,000,000 total costs by Active Award Count

Did not serve in past five yearsServed at least once in past five yearsPotential application can be reviewed

Num

ber o

f rev

iew

ers

Num

ber o

f app

licati

ons

potential reviewers: 3446 * .80 = 2757

additional applications: 5514

24

AcknowledgementsDivision of Statistical Analysis and Reporting http://grants.nih.gov/grants/oer_offices/subopac_dsar.htm

Robin Wagner, Ph.D.Matthew EblenDeepshika RoychowdhurySam Shuptrine

Division of Planning and EvaluationJoy Wang, Ph.D. Pritty Joshi, Ph.D.

Research Triangle Institute, International