nih loan repayment program evaluation milton j. hernández, ph.d. director division of loan...

NIH LOAN REPAYMENT PROGRAM EVALUATION

Milton J. Hernández, Ph.D.

Director

Division of Loan Repayment

OEP, OER, OD

National Institutes of Health

Bethesda, MD

NIH Loan Repayment ProgramsExtramural LRPs

FY 2003-2009

Laurel Haak, Ph.D.

Chief Science Officer

Discovery Logic, Inc.

Rockville, MD

2

NIH LRP Evaluation Questions

Question 1: Who is applying to the NIH LRP?

Question 2: Does participation in the LRP affect career outcomes?

Question 3: How does the LRP compare to other NIH career development award programs?

NIH Loan Repayment Programs

3

New and Renewal LRP Applications and Awards

Trends Analysis, FY03-FY07

NEW RENEWAL


The renewal program has decreased the success rate for new applications but has not affected the number of new

applications submitted. The success rate for new applications from FY05-FY07 is 40% and for renewals is 70%.

The renewal program has decreased the success rate for new applications but has not affected the number of new

applications submitted. The success rate for new applications from FY05-FY07 is 40% and for renewals is 70%.

4

Gender Distribution of New LRP Applicants



The ratio of women to men LRP

applicants is reversed from the

ratio in recent graduating MD and

PhD classes.

Success rates are not different.

The ratio of women to men LRP

applicants is reversed from the

ratio in recent graduating MD and

PhD classes.

Success rates are not different.

5

Gender Distribution of New LRP Applicants, by Degree



Degree Type Example Degrees

Physician Doctorate MD, DO

Academic Doctorate PhD, ScD, EdD, DPH

Physician Doctorate / Academic Doctorate MD/PhD, DO/ScD

Professional Clinical Doctorate

DDS, DMD, DVM, PharmD, PsyD

Allied Health Professional BSN, MPH, MSN

About the same number of men and

women applicants are MDs, but 2.5

times more women applicants are

PhDs, and twice as many men

applicants hold dual degrees.

About the same number of men and

women applicants are MDs, but 2.5

times more women applicants are

PhDs, and twice as many men

applicants hold dual degrees.

p<0.001

p<0.001

6

Race/Ethnicity of New LRP Applicants



More Blacks and fewer Asians applied as compared to recent MD and PhD graduating classes.

Success rates did not differ from each other or from the overall success rate.

More Blacks and fewer Asians applied as compared to recent MD and PhD graduating classes.

Success rates did not differ from each other or from the overall success rate.

7Prepared by NIH OER/OEP/DLR

Educational Debt, by Degree Type


Among LRP applicants, academic

doctorates have a higher debt load

than recent PhD graduates, and

physician doctorates have a lower

debt load than recent MD graduates.

Among LRP applicants, academic

doctorates have a higher debt load

than recent PhD graduates, and

physician doctorates have a lower

debt load than recent MD graduates.

†Source: Survey of Earned Doctorates, National Science Foundation (2003).*Source: Survey of Graduates, American Association of Medical Colleges (2003-2007).

8

Years Since Degree



LRP attracts early career researchers

72% of new applicants apply within 6

years of receiving their degree



LRP attracts early career researchers





Academic doctorates and physician doctorates apply at different times

67% of academic doctorates apply within 3 years of their degree

68% of physician doctorates apply 4-6 years after receiving their degree

Academic doctorates and physician doctorates apply at different times

67% of academic doctorates apply within 3 years of their degree

68% of physician doctorates apply 4-6 years after receiving their degree

4-6 Years(37%)

Up to 3 Years(35%)

7-9 Years(18%)

10+ Years(10%)

9

NIH LRP Evaluation: Outcome Analysis

Question 1: Who is applying to the NIH LRP?

Question 2: Does participation in the LRP affect career outcomes?

Question 3: How does the LRP compare to other NIH career development award programs?


10

Outcome Analysis Methodology

Outcome analysis: Regression Discontinuity Analysis.

We compared two nearly-identical applicant groups. LRP applicants with very similar priority scores were identified, about half of whom were funded and half of whom were not funded.


Not FundedNot FundedFunded

Funded

Priority Score “Bubble”

Priority Score “Bubble”

Identifying the Comparison Cohort

11NIH Loan Repayment Programs

F8NF8

NIAID, FY05 F9

NF9

NIDDK, FY04

F11NF14

NICHD, FY04

Identify bubble applicants for

each IC and each fiscal year

Identify bubble applicants for

each IC and each fiscal year

F259

NF254

IC-FY Bubble513

Combineindividualbubbles

Priority Scorerange,

e.g. 190-214

F8

NF9

NIMH, FY03

Funded

NotFunded

A 50-50 mix of funded and not funded applicants with closely-matched priority scores

Outcome AnalysisSubsequent publications

and NIH grant participation

Outcome AnalysisSubsequent publications

and NIH grant participation

LRP Applicant Cohort, FY03-FY06

IC-FY Bubble, Demographic Distribution

Parameter Category

%Funded

IC-FY Bubble(n=259)

%Not Funded IC-FY Bubble

(n=254)

%IC-FY Bubble

(n=513)

% All New

Applicants, FY03-FY07(n=9820)

GenderMale 41% 36% 38% 44%

Female 57% 62% 60% 53%

Degree Type

Physician Doctorate 48% 46% 47% 47%

Professional Clinical Doctorate 2% 6% 4% 5%

Academic Doctorate 44% 45% 45% 42%

Physician Doctorate / Academic Doctorate 5% 2% 3% 6%

Allied Health Professional 1% 1% 1% <1%

Years Since Degree

Up to 3 Years 52% 57% 54% 35%

4-6 Years 34% 35% 35% 37%

7-10 Years 11% 6% 8% 18%

10+ Years 2% 3% 3% 10%

12

Outcome Analysis, LRP Applicant Cohort, FY03-FY06

The demographics of the funded and not funded IC-FY Bubble cohorts are indistinguishable, hence they can be used as comparison groups. There are some notable differences compared

to the total new applicant group.

The demographics of the funded and not funded IC-FY Bubble cohorts are indistinguishable, hence they can be used as comparison groups. There are some notable differences compared

to the total new applicant group.

13

p=0.0102

NIH Grant Participation: IC-FY Bubble Overall

NIH Grant Participation: IC-FY Bubble, by Gender

Outcome Analysis, LRP Applicant Cohort, FY03-FY06

Male Female


Women LRP awardees are less likely than men to have applied for an NIH award or to have a subsequent observable role in the NIH extramural program.

Women LRP awardees are less likely than men to have applied for an NIH award or to have a subsequent observable role in the NIH extramural program.

p=0.0474

p=0.0147

NIH Grant Participation: IC-FY Bubble, by Degree

Outcome Analysis, LRP Applicant Cohort, FY03-FY06Physician Doctorate Academic Doctorate


Participation in the LRP program increased the likelihood that a physician doctorate applied for a subsequent NIH award and was retained in the NIH extramural workforce. It also increased the

likelihood that an academic doctorate was awarded an NIH grant.

Participation in the LRP program increased the likelihood that a physician doctorate applied for a subsequent NIH award and was retained in the NIH extramural workforce. It also increased the

likelihood that an academic doctorate was awarded an NIH grant.

p=0.0106

p=0.0470

p=0.0324

Question 3: Comparison with Other NIH Programs


LRP CohortLRP Cohort

Outcome analysisSubsequent NIH

grant participation

Researchaccomplishments

Publications

Demographicparameters

Identification of Comparison Cohorts

LRP Top-10%n = 215

LRP Top-10%n = 215

K Top-10%n = 218

K Top-10%n = 218

K CohortK01, K08, K23

K CohortK01, K08, K23 LRP+K CohortLRP+K Cohort

LRP+K Top-10% n = 96

LRP+K Top-10% n = 96

Funded Type I awardees, FY03-FY06Top 10% priority score, each FY/IC

L awards only


K awards only


L and K awards*

*L award between FY03-FY06 anda K01/K08/K23 award at any time. Awardees were in the top 10% of priority scores for both award mechanisms.

NIH Grant Participation: K and L Programs

Outcome Analysis, 0-6 Years Since Degree, Top-10% Comparison


Even after balancing for years since degree, the LRP recipients show less likelihood of applying for subsequent NIH funding than K or LRP+K awardees. LRP awardees are also less likely than K awardees to have a subsequent role on NIH awards.

Even after balancing for years since degree, the LRP recipients show less likelihood of applying for subsequent NIH funding than K or LRP+K awardees. LRP awardees are also less likely than K awardees to have a subsequent role on NIH awards.

p=0.0003

p=0.0176

p=0.0003

nih loan repayment program evaluation milton j. hernández, ph.d. director division of loan...

Documents

new applicants

men lrp applicants

renewal lrp applications

women applicants

degreetrends analysis

nih oeroepdlreducational

awardstrends analysis

physician doctorates