magnet schools and peers: effects on student achievement dale ballou vanderbilt university november,...

Magnet Schools and Peers: Effects on Student Achievement

Dale Ballou

Vanderbilt University

November, 2007

Thanks to Steve Rivkin, Julie Berry Cullen, Adam Gamoran, Ellen Goldring and Keke Liu.

Research Questions

• Has attending a magnet school caused an increase in mathematics achievement?

• How large is the influence of peers on mathematics achievement?

• How much of the magnet school effect remains after controlling for the influence of peers?

Study Setting

• Middle Schools in a Large Southern District

1 selective academic magnet

4 non-selective magnets

5 student cohorts

6 years: 1998-99 through 2003-04

Grades 5 & 6

Admissions Lotteries

• Oversubscribed magnets conduct lotteries

• Students may enter multiple lotteries

• Students who are not outright winners are placed on wait lists

• Wait-listed students accepted until the first week of school

Non-lottery Admissions

• Sibling preferences• Promotion from a feeder school• Geographic priority zone

These students are not included in the study sample (though they do enter the calculation of peer characteristics).

Research Design

• Lotteries assign students randomly to school type and to peers (magnet school peers vs. non-magnet peers).

• Randomized design circumvents biases arising from self-selection of schools and peers.

Limitations of Design

• Results may not generalize beyond lottery participants.

• Effects are relative (magnet schools vs. mix of non-magnet schools attended by lottery losers).

Lottery Participation Academic Composite

Non-Academic

Applicants 2315 2594

Outright Winners

883 1450

Delayed Winners

223 756

Losers,This Lottery

1209 388

Losers, All Lotteries

539 199

Grade 5 Enrollments

Academic Composite Non-Academic

This Magnet 758 1061

Other Magnets 287 339

Non-Magnets 834 846

Left system or Not Tested

436 346

• Substantial non-compliance, especially among winners of non-academic lotteries, attenuates estimated treatment & peer effects based on comparison of winners and losers.

Remedy: Use lottery outcomes as instruments to predict probability of attending magnet school, outcomes interacted with peer variables at magnet & zoned schools as instruments for peer characteristics.

Potential Pitfalls

• High rates of attrition from district can introduce systematic differences between treatment and control groups.

Remedies:Control for student characteristics (race, income, ESL, special ed, gender, prior achievement).Analyze attrition patterns for evidence of differences between winners and losers.

• Participating in multiple lotteries increases chances of winning. “Multiple participants” may differ in ways related to achievement.

Remedy: Control for the combination of lotteries each student entered. Winners are compared to losers who entered the same combination.

• Lotteries randomly assign students to magnet school peers or peers in their neighborhood (zoned) school, but lotteries do not determine the characteristics of the latter—residential decisions do.

Remedy: Control for characteristics of the peers in the zoned school.

Peer Characteristics

• Percentages black, low income (free & reduced-price lunch program), special ed, ESL, female

• Absenteeism rate

• Disciplinary incidents (rate per student)

• Intra-year mobility

• Prior achievement in math and reading

Model (Summary)

• Two treatment variables (academic magnet, composite non-academic magnet)

• Variation in peers resulting from lottery outcomes

• Other controls (student characteristics, peers at the zoned school, lottery participation indicators, year by grade effects)

Findings

• When model does not include peer characteristics

- Academic magnet, + 18% in grade 5, drops to +10% in grade 6 (% of normal year growth)

- Non-academic magnet, no grade 5 effect, +54% in grade 6

• When models include peer characteristics- Reducing percent black from 75% to 25% increases scores by 60% of normal year growth.- Effect of percent low income is about half that large.- Other peer characteristics have no statistically significant effect.

- Controlling for either percent black or percent low income, the effect of the academic magnet disappears.

- The large 6th grade effect in the non-academic magnets remains substantially undiminished.

Checking Alternative Interpretations

• Are peers a proxy for heterogeneous response to treatment?

Check: Interact magnet treatment indicators with all observed student characteristics.

Finding: Peer effects are undiminished.

• Are peers a proxy for teacher quality?

Check: Control for teacher quality by including teacher fixed effects.

Finding: Peer effects are undiminished.

Attrition, Academic Magnet

Lottery Participants

Left System After Grade: Winners Losers

4 13% 21%

5 8% 14%

6 9% 11%

7 6% 9%

Attrition, Composite Non-Academic Magnet

Lottery Participants

Left System After Grade: Winners Losers

4 8% 12%

5 12% 9%

6 10% 4%

7 12% 16%

Potential Attrition Biases

• Lottery losers are more likely to leave the system than winners.

• Losers are also more likely to leave when they can afford private schooling. These tend to be higher-achieving students.

Result: Losers who remain in the system have lower achievement than winners who remain.

• Unfavorable peers at zoned school make losers more likely to leave system.

• Effect greatest among those who can afford private schooling.

Result: Quality of peers positively correlated with losers’ achievement. Estimated peer effects appear too strong.

Checking Attrition Bias

• Are rates of attrition correlated with variables that predict individual achievement (race, income, prior achievement)?

• Yes, but not differently for winners and losers.

Conclusions

• For at least some students in some places, magnet schools have a positive effect on academic achievement.

• There are very strong peer effects on middle school achievement. Do not appear to operate through behaviors readily quantified with administrative data (attendance, disruptions, mobility).