Studying Variation in the Effect of Program Participation

Stephen W. Raudenbush

Presentation at the

“Workshop on Learning from Variation in Program Effects”

Palo Alto, July 19, 2016

The research reported here was supported by a grant from the W.T. Grant Foundation to the University of Chicago entitled “Building Capacity for Evaluating Group-Level Interventions.” Thanks to Sean Reardon and Takako Nomi for their collaboration on these ideas.

Outline1. Pervasiveness of

• Multi-site trials• Non-compliance

2. Instrumental variables in a single-site study• Under homogeneity of impact• Under heterogeneity of impact• Examples

3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses• Method 2: Two-stage generalized least squares• Method 2= “Between-Site Regression!”

4. Design Considerations

5. Modeling Program Participation and Program Impact on Participants

1. Pervasiveness of Multi-Site Trials

Since 2002, IES has funded 175 group-randomized trials

Vast majority are multi-site trials (Spybrook, 2013)

Other recent examples

* National Head Start Evaluation (US Dept of HHS, 2010)

* Moving to Opportunity (Sonbanmatsu, Kling, Duncan, Brooks Gunn, 2006)

* School-based lottery studies (Abdulkadiroglu, Angrist, Dynarski, Kane, and Pathak, 2009).

* Tennessee STAR (Finn and Achilles, 1990)

* Ending Social Promotion (Jacob and Lefgren, 2009)

* Double-Dose Algebra (Nomi and Allensworth, 2009)

* Welfare to Work (Bloom, Hill, Riccio, 2003)

* Small Schools of Choice (MDRC)

Some Recent MS TrialsStudy Levels Assigned

UnitsSites Fixed or

Random sites

National Head Start Eval.

2 Children 198 Program Sites

Random

Moving to Opportunity

2 Families 5 cities Fixed

BostonCharter School Lotteries

2 Children Lottery pools Random

Tennessee STAR

3 Teachers 79 Schools Random

4 R’s 3 Classrooms 18 Schools Random

Double-Dose Algebra

2 Children 60 Schools Random

2. Estimating the Impact of Program Participation in a One- Site Study

T=Random assignment M=Participation

Y=outcome

Figure 1: Conventional Instrumental Variable Model (Homogeneous Treatment Effects)

0/

:EffectITTTotal

so

Single site, heterogeneous treatment effects

TM

Y

Person-specific Causal Model

)(E

)(E

),()(

CovBEB iii

“No Compliance-Covariance” Assumption is Strong!

0/0),(

),()(

ifCovif

CovBE

Alternative Approachfor binary M

“Local Average Treatment Effect” (LATE)or“Complier Average Treatment Effect”

(Bloom, 1984; Angrist, Imbens, and Rubin, 1996)

Principal StratificationStratum M(1) M(0) Г=M(1)-M(0) Y(M(1))-Y(M(0)) Fraction

of popAverageEffect

Compliers 1 0 1 Y(1)=Y(0) γcompliers δcompliers

Always-takers

1 1 0 Y(1)-Y(1)=0 γ always 0

Never-takers

0 0 0 Y(0)-Y(0)=0 γnever 0

Defiers 0 1 -1 Y(0)-Y(1) 0 0

Complier-average treatment effect(“Local average treatment effect”)

randomizedisTifMonITTTMETME

YonITTTYETYENote

so

BE

complierscompliers

neveralwayscomplierscompliers

"")0|()1|("")0|()1|(

/

*0*0)(

In SumWe can estimate the Population-Average

Effect of Participating if we assume Cov(Г,Δ)=0

We can estimate LATE if we assumePr(Г≥0)=1

The latter is a weaker assumption, but does not eliminate the selection problem!

Multiple SitesHow do we take this to multiple sites to

* Estimate average Impact of Program Participation

* Estimate variation in the Impact of Program Participation

* Two methods using simulated data:“Small Schools of Choice” Design (J=200, 80<n<120)

)430.0,194.1:( 2 valuestrue

Method 1: Combine 2 ITT analyses

Step 1: Estimate the Impact of Treatment Assignment on the Outcome

Results

2)(,)(

.).(.

B

jijjijjjij

BVarBE

eeTTBYY

)75.1,21.(499.*96.1770.

forintervalvalueplausible%95

499.249.0ˆ53.0,770.0ˆ

22

j

B

B

se

Step 2: Estimate the Impact of Treatment Assignment on Program Participation

Results2)(,)(

.).(.

G

jijjijjjij

GVarGE

TTGMM

)86,55(.

forintervalvalueplausible%95

078.0061.0ˆ703.0ˆ

22

j

G

G

Step 3: Combine Results

)248.1,267.(483.*96.1095.1)(%95

483.)703(.)078(.

)078(.*)095.1()249(.ˆ

caseourIn

095.1703./770.ˆ/ˆˆ/

22

222

22

2222

DPV

DGif

caseourIn

D

D

In sumTrue Values

Our estimates

430.0,19.1: 2 valuesTrue

487.0ˆ,10.1ˆ 2

Method 2: Two-Stage Generalized Least Squares: Theoretical Model

2

2

)(,)(

.).(.:2Stage

)(,)(

.).(.:1Stage

D

jijjijjjij

G

jijjijjjij

DVarDE

MMDYY

GVarGE

TTGMM

Method 2 in Practice

2

**

2

**

)(,ˆ)(

).(ˆ.

originalusingvariancesRecompute.3

)(,)(

).ˆ(.:.2

).(ˆ.ˆ:.1

D

ijijjijjjij

D

ijijjijjjij

jijOLSjij

DVarDE

MMDYY

M

DVarDE

MMDYYHLMDo

TTGMMCompute

Method 2=“Between Site Regression!”

Cj

Ejjjjj

jjj

jjj

GDGB

whichfrom

GDG

GDB

ˆ)(ˆˆ.....

)(

0)](ˆ[ jj DGERequires

Results

)863.,562(.)746.,316(.%95

6968.

4856.ˆ

0730.0,095.1ˆ

22

2

2

CI

se

Envisioning Variation:LATE Effect

“Head Start” Design (J=200, 10<n<20)

“Small Schools of Choice” Design (J=200, 80<n<120)

“Welfare to Work” Design (J=60, 200<n<1400)

Program Participation Model (“LATE”)

)430.0,194.1:(

280.0ˆ,007.1ˆ:

487.0ˆ,095.1ˆ0ˆ986.0ˆ:

),(~

.).(.:

)(,)(

.).(:

2

2

2

2

2

2

valuestrue

WtW

SSC

HS

ND

MMDYYModelImpact

GVarGE

vvTTGMMmodelionParticipat

j

jijjijjjij

jijjijjjij

Profile Likelihood for LATE: “HS” Design

0 0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01-1.00

0

1.00

2.00

3.00

Tau

Bet

a

Profile Likelihood for LATE: “SSOC” Design

0 0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01

-0.50

1.00

2.50

Tau

Bet

a

Profile Likelihood for LATE: “WtW” Design

0 0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01-1.00

0

1.00

2.00

3.00

Tau

Bet

a

Posterior intervals for site-specific LATE Effects

Posterior Intervals for LATE: “HS” Design

-0.55

0.27

1.09

1.90

2.72

MH

AT

0 50.50 101.00 151.50 202.00

Posterior Intervals for LATE: “SSC” Design

0 50.50 101.00 151.50 202.00-1.21

-0.13

0.96

2.04

3.12

MH

AT

Posterior Intervals for LATE: “W to W” Design

0 15.50 31.00 46.50 62.00-0.37

0.41

1.18

1.95

2.73

MH

AT

Moving Toward Explanationmodeling participation, modeling impact

Total Impact of Assignment=Impact of Assignment on Participation * Impact of participation on Outcome

Within site:

Which persons are most likely to participate?Which persons are most likely to benefit from participation?

Between Sites:

How do we improve site-average participation rate?How do we enhance average benefit of participating?

Models are needed at both levels because sites vary not only in organizational effectiveness but also in client composition

The picture can't be displayed.

jjj DGB