Studying the Mean and Variation in the Effect of Program Participation

in Multi-site Trials

The research reported here was supported by a grant from the W.T. Grant Foundation to the University of Chicago entitled “Building Capacity for Evaluating Group-Level Interventions.” Thanks to Takako Nomi for her collaboration on these ideas.

Outline1. Pervasiveness of

• Multi-site trials• Non-compliance

2. Potential outcomes framework • Observed data• Potential outcomes• Causal effects

3. Instrumental variables in a single-site study

• Under homogeneity of impact• Under heterogeneity of impact• Complier-Average Causal Effect (CACE)

4. Instrumental variables in multi-site studies

– Estimating the Average CACE– Estimating the Variation in CACE

1. Non-compliance in Multi-Site Trials

Examples

* National Head Start Evaluation (US Dept of HHS, 2010)

* Moving to Opportunity (Sonbanmatsu, Kling, Duncan, Brooks Gunn, 2006)

* School-based lottery studies (Abdulkadiroglu, Angrist, Dynarski, Kane, and Pathak, 2009).

* Tennessee STAR (Finn and Achilles, 1990)

* Double-Dose Algebra (Nomi and Allensworth, 2009)

* Small Schools of Choice (MDRC)

2. Observed Data (Single Site)

Observed variables

outcomeY

notif

treatmentineparticipatifM

controltoassignedif

treatmenttoassignedifT

i

i

i

0

1

0

1

2. Potential Outcomes and Causal Effects

YonMofimpactMYM

MonTofimpactMM

YonTofimpactYY

i

iii

iii

)0()1(

)0()1(

)0()1(

2. Average Causal Effects

)(

)(

)(

i

i

i

i

i

i

E

E

E

effectscausalAverage

YonMofimpact

MonTofimpact

YonTofimpact

T=Random assignment M=Participation

Y=outcome

Conventional Instrumental Variable Model (Homogeneous Treatment Effects)

0/

:EffectITTTotal

so

What happens if impacts are heterogeneous?

Single site, heterogeneous treatment effects

TM

Y

Person-specific Causal Model

)(E

)(E

/),(/

),()(

Cov

CovBE

B iii

Assume away “Compliance-Effect Covariance”??

Alternative Approachfor binary M

“Complier Average Causal Effect” (CACE)

or

“Local Average Treatment Effect”

(Bloom, 1984; Angrist, Imbens, and Rubin, 1996)

Principal Stratification(Frangakis and Rubin, 2002)

Stratum M(1) M(0) Ф=M(1)-M(0) Y(M(1))-Y(M(0)) Fraction of pop

Average

Effect

Compliers 1 0 1 Y(1)-Y(0) πcompliers δcompliers

Always-takers

1 1 0 Y(1)-Y(1)=0 πalways 0

Never-takers

0 0 0 Y(0)-Y(0)=0 πnever 0

Defiers 0 1 -1 Y(0)-Y(1) 0 0

Complier-average causal effect

compliers

compliers

complierscompliers

neveralwayscomplierscompliers

so

BE

/

*0*0)(

In Sum

We can estimate the Population-Average Effect of Participating if we assume Cov(Φ,Δ)=0

We can estimate CACE if we assumePr(Φ ≥0)=1

The latter is a much weaker assumption

2. Causal Effects in Multi-site Trials

,,meansOverall

,,meansSpecificSite

:SpecificPerson

jjj

ij

ij

ij

YonMofimpact

MonTofimpact

YonTofimpact

TM

Y

Site-specific Causal Model

jjj

j

j

Multiple Sites:Causal Effects

)430.0,194.1:( 2 valuestrue

2)(,)(

.).(.

jj

jijjijjjij

VarE

MMYY

Combine 2 ITT analyses

Step 1: Estimate the Impact of Treatment Assignment on the Outcome

Results

111

1

)(,)(

.).(.

VarE

eeTTYY jijjijjjij

)75.1,21.(499.*96.1770.

forintervalvalueplausible%95

499.249.0ˆ

53.0,770.0ˆ

1

2211

1

j

se

Step 2: Estimate the Impact of Treatment Assignment on Program Participation

Results

2

1

)(,)(

.).(.

VarE

TTMM jijjijjjij

)86,55(.

forintervalvalueplausible%95

078.0061.0ˆ

703.0ˆ

22

jG

Step 3: Combine Results: mean

095.1703./770.ˆ/ˆˆ

/

caseourIn

Step 3: Combine Results: variance

)248.1,267.(483.*96.1095.1)(%95

483.)703(.)078(.

)078(.*)095.1()249(.ˆ

caseourIn

)(

22

222

22

2222

222222

jPV

so

if

In sum

True Values

Our estimates

430.0,19.1: 2 valuesTrue

487.0ˆ,10.1ˆ 2