The Impact of Postsecondary Remediation Using a

Regression Discontinuity Approach:

Addressing Endogenous Sorting and

Noncompliance

Juan Carlos CalcagnoMathematica Policy Research, Inc.

IES Research ConferenceJune 11th, 2008

Juan Carlos Calcagno, June 11th, 2008 2 :: 12

>> Introduction: Why is this Important?

Little is known about the causal effects of remediation on student outcomes:» Lack of data

» Nonrandom selection into remediation

Controversial topic in postsecondary education:

» Remediation increases access but is costly to states and colleges

» Should we pay twice for academic preparation that should have

occurred in secondary school?

» Large proportion of entering students requiring remedial courses

» Remedial credits do not count toward degrees, but students pay

tuition for these courses

Juan Carlos Calcagno, June 11th, 2008 3 :: 12

Estimate the causal impact of remediation in Florida community colleges using a unique administrative dataset.

Use a quasi-experimental design (regression discontinuity) based on a remedial placement cutoff rule to solve the selection problem.

Discuss threats to internal validity and implications for practice: noncompliance and endogenous sorting

>> Contribution of this Study

Juan Carlos Calcagno, June 11th, 2008 4 :: 12

>> FL Administrative Dataset

All first-time, degree-seeking community college students

starting in the Fall of 1997 to 2000 (over 100,000 records)

Term-by-term transcript data through Spring of 2006

All test scores reported (CPT, SAT, ACT) plus other controls

Outcomes: passing Algebra/English 101, retention,

certificate, associate degree, transfer to state 4-year college,

credits earned (remedial and non-remedial)

Juan Carlos Calcagno, June 11th, 2008 5 :: 12

>> The Regression Discontinuity Design: Intuition

Beth and Becky are observationally similar

Compare the outcomes of Beth and BeckyCompare the outcomes of Beth and Becky

Beth and Becky both take the College Placement Test (CPT)

Beth scores just above the cut-off score

Becky scores just below the cut-off score

Crossover: Beth take remediation

No-show: Becky never enroll in remediation

Beth go to college-level courses Becky go to remediation

Endogenous Sorting: Beth

retests to place out of remediation

Juan Carlos Calcagno, June 11th, 2008 6 :: 12

>> Noncompliance

Probability of Enrollment in Remediation by CPT Score and Subject

-50 -40 -30 -20 -10 0 10 20 30

CPT Score Relative to Math Cutoff-50 -40 -30 -20 -10 0 10 20 30

CPT Score Relative to Reading Cutoff

-50 -40 -30 -20 -10 0 10 20 30

CPT Score Relative to Writing Cutoff

`

CPT Score Relative to Math Cutoff CPT Score Relative to Reading Cutoff

Juan Carlos Calcagno, June 11th, 2008 7 :: 12

-50 -40 -30 -20 -10 0 10 20 30

>> Endogenous Sorting around Cutoff

Density of Reading CPT for Institution E

0.0

1.0

2.0

3.0

4

-50 -40 -30 -20 -10 0 10 20 30

Juan Carlos Calcagno, June 11th, 2008 8 :: 12

>> The Econometric Model

D: assignment to remediation (below cutoff)

is the intention-to-treat estimator (ITT)

Z: CPT test score

Robustness tests: (1) add controls; (2) narrow sample; (3) endogenous sorting; (4) functional form

& standard errors

iiii )Z(fDy ε+δ+β+α= 1

1β

is the local average treatment effect (RD-IV)

T: enrollment in remedial education

2βiiii )Z(fT̂y ε+δ+β+α= 2

iiii v)Z(fDT +γ+δ+α= 1

Juan Carlos Calcagno, June 11th, 2008 9 :: 12

ITT RD-IV ITT RD-IV ITT RD-IV ITT RD-IV

(1) (2) (3) (4) (5) (6) (7) (8)

-0.014 -0.022 -0.018 -0.030 0.006 0.012 -0.001 -0.002

(0.012) (0.020) (0.011) (0.019) (0.029) (0.057) (0.031) (0.064)

0.020 0.035 0.014 0.026 0.004 0.008 0.007 0.015

(0.012) (0.021) (0.011) (0.019) (0.032) (0.062) (0.025) (0.051)

-0.004 -0.006 -0.003 -0.006 -0.004 -0.008 -0.007 -0.014

(0.002) (0.004) (0.002) (0.004) (0.005) (0.009) (0.006) (0.012)

-0.006 -0.010 -0.006 -0.011 -0.016 -0.032 -0.014 -0.027

(0.006) (0.011) (0.006) (0.011) (0.016) (0.031) (0.012) (0.025)

-0.001 -0.002 -0.003 -0.005 -0.022 -0.043 -0.033 -0.067

(0.006) (0.010) (0.006) (0.010) (0.017) (0.033) (0.018) (0.037)

3.590** 6.169** 3.290** 5.690** 3.797* 7.453* 3.515* 7.282*

(0.657) (1.099) (0.613) (1.023) (1.698) (3.425) (1.621) (3.252)

0.233 0.400 0.011 0.019 1.398 2.744 -0.118 -0.244

(0.649) (1.113) (0.596) (1.031) (1.836) (3.622) (1.759) (3.641)

Institutions 28 28 28 28 28 28 19 19

Observations 96,724 96,724 96,724 96,724 14,493 14,493 9,593 9,593

Without Controls With Controls

All students

Transfer to State University System (SUS)

Total Credits Earned

Total Non-Remedial Credits Earned

Fall-to-Fall Persistence

Earning a Certificate

Associate Degree Completion

No-Retesting & Narrow Band Sample

Completion of First College-Level Course

Narrow Band Sample

>> The Impact of Math Remediation

Juan Carlos Calcagno, June 11th, 2008 10 :: 12

ITT RD-IV ITT RD-IV ITT RD-IV ITT RD-IV

(1) (2) (3) (4) (5) (6) (7) (8)

-0.066** -0.095** -0.060** -0.086** -0.053** -0.090** -0.028* -0.036*

(0.008) (0.012) (0.008) (0.012) (0.009) (0.017) (0.013) (0.017)

-0.009 -0.012 -0.003 -0.003 -0.017 -0.029 -0.009 -0.013

(0.008) (0.011) (0.008) (0.011) (0.010) (0.017) (0.018) (0.028)

-0.002 -0.003 -0.002 -0.002 -0.004 -0.007 -0.003 -0.005

(0.002) (0.003) (0.002) (0.003) (0.004) (0.007) (0.005) (0.008)

-0.025** -0.037** -0.020** -0.029** -0.024** -0.040** -0.020 -0.031

(0.004) (0.006) (0.004) (0.006) (0.009) (0.014) (0.017) (0.026)

-0.016** -0.024** -0.009* -0.013* -0.015* -0.025* -0.004 -0.005

(0.004) (0.005) (0.004) (0.006) (0.007) (0.011) (0.016) (0.022)

1.527** 2.266** 2.048** 3.025** 0.854 1.437 1.858 2.889

(0.447) (0.647) (0.461) (0.653) (0.496) (0.818) (1.158) (1.740)

-1.751** -2.599** -1.190** -1.758** -1.182 -2.159 -0.935 -1.590

(0.467) (0.685) (0.431) (0.636) (0.684) (1.271) (1.252) (2.124)

Institutions 28 28 28 28 28 28 7 7

Observations 97,938 97,938 97,938 97,938 37,747 37,747 8,775 8,775

No-Retesting & Narrow Band Sample

Completion of First College-Level Course

Narrow Band SampleWithout Controls With Controls

All students

Transfer to State University System (SUS)

Total Credits Earned

Total Non-Remedial Credits Earned

Fall-to-Fall Persistence

Earning a Certificate

Associate Degree Completion

>> The Impact of Reading Remediation

Juan Carlos Calcagno, June 11th, 2008 11 :: 12

>> Summary of Findings

Math remediation increases persistence to 2nd year (2 to 3.8%)

for students on the margin of passing the cutoff.

Impacts on total credits earned are positive (7 and 3 points, M

& R), but not different from zero for college-level credits.

No statistical differences for all other outcomes

The likelihood of passing English 101 was slightly lower for

reading remedial students while no difference was found in

passing Algebra 101 for math remedial students.

Juan Carlos Calcagno, June 11th, 2008 12 :: 12

>> Policy Implications

Remediation might promote early persistence in college, but it

does not necessarily help students on the margin of passing the

cutoff to make progress toward a degree.

Costs of remediation should be given careful consideration in

light of the limited benefits.

State Departments of Education should explore noncompliance

and retesting practices and consider potential consequences.