atila abdulkadiro glu (duke) nikhil agarwal (mit and nber) · nikhil agarwal (mit and nber) parag...

The Welfare Effects of Coordinated Assignment:Evidence from the NYC HS Match

Atila Abdulkadiroglu (Duke)

Nikhil Agarwal (MIT and NBER)

Parag A. Pathak (MIT and NBER)

April 2015

1/30

Introduction

Motivation

Public school choice is a growing part of contemporary educationreform movement’s embrace of quasi-market reforms

Recent efforts towards coordinated and centralized choice

X Standardized application timelines

X Common application forms (paper and/or online)

X Integration of traditional district and charter school admissions

X Multiple vs. single offer systems

X Student demand and school supply matched using algorithm

X Centrally managed waitlists and enrollment

2/30

Introduction

In 2012, new unified enrollment systems adopted in Denver and NewOrleans; in 2014, Newark and Washington DC

Choice reforms controversial due to zero-sum nature

I Some inevitably end up with better schools, while others do not

Change in NYC High School mechanism in 2003 provides uniqueopportunity to study consequences of coordinated and centralizedchoice

Question: How did the new NYC student assignment mechanism affectthe allocation of students to schools and welfare from the assignment?

3/30

Introduction Background

Public High School Choice in New York

80,000 aspiring high school students, ∼600 programs at 200+ schools

2002-03: Common application system with uncoordinated offers andacceptances

X Students could receive multiple offers

2003-04: Centralized admissions coordinated through a variant ofstudent-proposing deferred acceptance algorithm

X Students received a single assignment

Supplementary/administrative processes place unassigned students

Copycats: 2005 Pan London Admissions scheme places 100,000 pupilsusing “equal preferences”

4/30

Introduction

Research Objectives

Describe offer processing, assignment, and enrollment patterns

Estimate distribution of student preferences for schools using rankorder lists submitted in new mechanism

Use estimated preference distribution to compare mechanisms andevaluate design tradeoffs

5/30

Introduction

Research Approach

Exploit sudden change to isolate the impact of the mechanismI Advertised widely in September, preferences submitted November 2003X Limited opportunity for participants to change residences in response

Use rich micro-level data on choices, assignment, and enrollmentI Allows for a demand model incorporating rich heterogeneityX Essential for evaluating the welfare effects of assignment changesX New mechanism’s straightforward incentive properties motivates

empirical approach based on (revealed) choices of students

Welfare approach focuses on student preferences and allocative issuesI Systematic data on preferences, offers and assignments from an

uncoordinated mechanism has been hard to obtainI Preference estimates can be used to evaluate alternative mechanisms,

quantifying design tradeoffs

6/30

New York City Mechanisms

Main differences between mechanisms

1 Ability to apply to more schools (12 vs. 5)

2 Single vs multiple offers

3 Sequential student response to school offersI The uncoordinated mechanism did not automatically reject lower ranked

offers for students with multiple offersX Only three rounds of offer processing as opposed to a computerized offer

processing system

4 Relaxation of strategic pressure on participants

X Advice in the old mechanism: “Determine what your competition is for aseat in a program.” (HS Directory 2002)

X Advice in new mechanism: “You must now rank your 12 choicesaccording to your true preferences”

7/30

Comparing Assignments and

Offer Processing

Assignment Enrollment(1) (2) (3)

Overall 70,358 3.36 3.50

First Round 23,867 4.23 4.11Second Round 5,780 4.55 4.44Third Round 4,443 4.35 4.26Supplementary Round 10,170 4.61 4.37Administrative Round 26,098 1.64 2.11

No Offers 36,464 2.80 3.12One Offer 21,328 3.89 3.85Two or More Offers 12,566 4.07 4.03

Overall 66,921 4.05 3.91

Main Round 54,577 4.02 3.86Supplementary Round 5,201 5.10 4.90Administrative Round 7,143 3.50 3.52

A. Uncoordinated Mechanism -‐ By Final Assignment Round

B. Uncoordinated Mechanism -‐ By Number of First Round Offers

C. Coordinated Mechanism -‐ By Final Assignment Round

Distance to School (in miles)Number of Students

Table 3. Offer Processing across Mechanisms

8/30

(1) (4) (5)

Overall 70,358 8.5% 18.6%

First Round 23,867 5.2% 9.6%Second Round 5,780 4.8% 11.4%Third Round 4,443 4.9% 14.2%Supplementary Round 10,170 7.8% 25.4%Administrative Round 26,098 13.3% 26.8%

No Offers 36,464 10.4% 24.4%One Offer 21,328 7.1% 13.8%Two or More Offers 12,566 5.7% 9.8%

Overall 66,921 6.4% 11.4%

Main Round 54,577 6.1% 9.9%Supplementary Round 5,201 4.8% 10.4%Administrative Round 7,143 9.6% 23.6%

Table 3. Offer Processing across Mechanisms

B. Uncoordinated Mechanism -‐ By Number of First Round Offers

A. Uncoordinated Mechanism -‐ By Final Assignment Round

C. Coordinated Mechanism -‐ By Final Assignment Round

Enrolled in School Other than Assigned

Exit from NYC Public SchoolsNumber of Students

9/30

School Access: Main Round

Ranked Schools Assigned Ranked Schools Assigned(1) (2) (3) (4)

Distance (in miles) 4.82 4.30 5.10 4.00

High Math Achievement 12.4 11.7 13.0 10.7High English Achievement 20.9 20.2 22.1 19.1Percent Attending Four Year College 49.1 47.1 50.6 48.3Fraction Inexperienced Teachers 45.3 45.6 46.6 43.8Attendance Rate (out of 100) 85.1 84.6 85.7 83.8Percent Subsidized Lunch 60.0 60.5 57.6 56.7Size of 9th grade 694.3 698.8 675.0 819.2Percent White 15.1 14.7 16.7 17.8

Table 4. Ranked vs. Assigned Schools by Student Assignment RoundUncoordinated Mechanism Coordinated Mechanism

A. Main Round

10/30

School Access: Supplementary Round





B. Supplementary Round

11/30

School Access: Administrative Round





C. Administrative Round

12/30

Naive Simulation

Two key features of the uncoordinated mechanism:

1) Limited rounds of offer processing2) Limited number of ranks

Holding fixed behavior, is it possible to decompose which feature isresponsible for large number unassigned after the main round?

X Approach: simulate main round of uncoordinated mechanism using 0304ranks

Ranks Rounds Unassigned

Full - 12 Unlimited 11,931Truncated - 5 Unlimited 15,468Full - 12 Three Rounds 34,257

13/30

Summary on Assignments and Offer Processing

Three rounds of offer processing left half the students unassignedI Second and third round processed only 10K students despite 17K

“extra” offers in the first round

In old mechanism, administrative round processed 36% of the studentsI Three times more children assigned in over-the-counter round in

uncoordinated mechanism (25K vs. 7K)I Administratively assigned students are more likely to exit and matriculate

elsewhere

Assigned schools are similar to ranked schools in the main andsupplementary rounds

I Significantly worse compared to ranked ones in administrative round

14/30

Summary on Assignments and Offer Processing

Travel distance increases by 0.69 miles post coordinationI Driven by changes in the administrative roundI For median applicant, closest school is 0.82 miles, but first choice is 3.51

miles ⇒ choice is valued

Aftermarket in the Uncoordinated mechanism is relatively flexibleI Coordinated mechanism: 11% enroll elsewhereI Uncoordinated mechanism: 19% enroll elsewhereX Distance to enrollment increases in administrative aftermarket in the

uncoordinated mechanism

15/30

Recovering Student

Preference Distribution

Demand Estimation Preference Gradients

3

3.5

4

4.5

5

5.5

6

6.5

7

7.5

8

1 2 3 4 5 6 7 8 9 10 11 12

Miles

Rank

Distance to Ranked School Program All

Low Baseline

High Baseline

Low Income

High Income

16/30


0

5

10

15

20

25

30

1 2 3 4 5 6 7 8 9 10 11 12

High Regen

ts M

ath Achievem

ent

Rank

School Math Achievement by Student Rank

All

Low Baseline

High Baseline

Low Income

High Income

17/30


Choice 1st 2nd 3rd 4th 5th 9th 12thN 69582 94% 89% 83% 77% 50% 20%

Mean Distance (miles) 4.43 4.81 5.05 5.21 5.38 5.65 5.12Median Distance (miles) 3.51 3.95 4.20 4.37 4.57 4.78 4.24

High English Achievement 27.8 25.6 24.3 23.2 22.2 19.9 18.6High Math Achievement 16.7 15.3 14.7 13.9 13.4 11.5 10.4Percent attending Four Year college 53.7 54.1 52.8 51.7 50.9 49.0 47.5Fraction Inexperienced Teachers 43.0 43.8 45.0 46.2 47.1 49.5 49.9Subsidized Lunch 51.4 53.4 54.5 56.2 57.4 61.3 63.1Attendance Rate 87.2 86.7 86.4 86.1 85.9 85.2 84.5

Percent Asian 13.8 13.7 13.3 12.7 12.2 10.1 9.4Percent Black 32.6 34.3 35.1 36.0 36.7 38.7 38.1Percent Hispanic 34.4 35.2 35.9 37.0 37.9 40.8 43.5Percent White 19.1 16.7 15.7 14.4 13.3 10.4 9.0

Table 6. School Characteristics by Student Choice in Coordinated Mechanism

X Distance, percent non-minority, and average test score monotone withrank

18/30

Demand Estimation Utility specification

A Cardinal Representation

Use distance as a numeraire for welfare (“willingness to travel”)I (Indirect) Utility of student i from program j given by:

uij = v(xj , zi , ξj , εij)− dij

X zi : student chars - e.g., race, subsidized lunch, baseline math andenglish, LEP, SPED, neighborhood income

X xj : program chars - e.g., size, performance, demos, program typesX ξj : school unobservablesX dij : distance between student i and program jX εij : individual-program specific taste variance

I Main assumption:εij ⊥ dij

∣∣ziX Unobserved tastes for schools may be correlated with residential

choicesX If students systematically reside near schools they prefer, we’d

underestimate the value of being near good schools

19/30

Demand Estimation Utility specification

Cardinal Representation

Exploit rich micro-data on student observables

uij = δj +∑`

α`z`i x`j +

∑k

γki xkj − dij + εij

δj = xjβ + ξj

I Implement an ordered version of Rossi, McCulloch, and Allenby (1996)’sMNP, which allows for Gibbs sampler through convenient choice ofdistributions, e.g.

γi ∼ N (0,Σγ) ξj ∼ N (0, σ2ξ) εij ∼ N (0, σ2

ε )

I Mixing allows for more flexible substitution patterns

Computation time is a serious constraint

Lots of experimentation with models within this family, and amongmultinomial logit models with broadly similar results

20/30

Demand Estimation Preference Estimates

-‐20.00 -‐15.00 -‐10.00 -‐5.00 0.00 5.00 10.00 15.00

Asian

Black

Hispanic

White

High Math Baseline

Low Math Baseline

Distance-‐equivalent u4lity (miles)

Willingness to Travel by Quar4le of Student U4lity

Top

Third

Second

BoGom

Mean utility across schools normalized to zero within each group details

21/30

Comparing Student

Welfare across Mechanisms

Welfare Alternative Assignments

Alternative Assignments

Compute the first-best optimum, by summing of (distance-metric)student utility subject to capacity constraints (utilitarian assignment)X Serves as benchmark of how much room left for students to improve

with changes to algorithm

1) Outcome produced by the student-proposing deferred acceptancealgorithm not student-optimal stable matching because schoolorderings not strict

X Computed using Erdil and Ergin (2008)’s stable improvement cyclesprocedure

2) Student-optimal stable matching can be dominated by a Paretoefficient matching

I Since not stable, schools that actively rank applicants could undermineassignment

Possible interpretations:1) “Cost of straightforward incentives for students”2) “Cost of stability for schools”

22/30


Assignment Mechanism:

(1) (2) (3) (4)All -‐18.96 -‐3.73 -‐3.62 -‐3.11Female -‐18.90 -‐3.71 -‐3.59 -‐3.07

Asian -‐18.08 -‐3.53 -‐3.43 -‐3.01Black -‐19.43 -‐3.89 -‐3.79 -‐3.25Hispanic -‐19.37 -‐3.80 -‐3.67 -‐3.10White -‐17.07 -‐3.21 -‐3.11 -‐2.82

Bronx -‐21.39 -‐4.63 -‐4.46 -‐3.72Brooklyn -‐18.48 -‐3.21 -‐3.14 -‐2.70Manhattan -‐20.07 -‐5.40 -‐5.25 -‐4.43Queens -‐18.02 -‐3.39 -‐3.29 -‐2.96Staten Island -‐13.82 -‐1.25 -‐1.10 -‐1.03

Table 8. Welfare Comparison of Alternative Mechanisms Compared to Utilitarian Assignment

Coordinated MechanismStudent-‐Optimal

MatchingOrdinal Pareto Efficient

Matching

School ChoiceNeighborhood Assignment

23/30


Assignment Mechanism:

(1) (2) (3) (4)High Baseline Math -‐18.53 -‐3.29 -‐3.18 -‐2.61Low Baseline Math -‐19.40 -‐4.28 -‐4.18 -‐3.63

Subsidized Lunch -‐19.16 -‐3.78 -‐3.66 -‐3.12Bottom Neighborhood Income Quartile -‐19.89 -‐4.25 -‐4.12 -‐3.46Top Neighborhood Income Quartile -‐17.44 -‐3.63 -‐3.51 -‐3.15

Special Education -‐19.41 -‐4.83 -‐4.73 -‐4.11Limited English Proficient -‐19.81 -‐3.74 -‐3.64 -‐3.16SHSAT Test-‐Takers -‐19.13 -‐4.17 -‐4.05 -‐3.41

Table 8. Welfare Comparison of Alternative Mechanisms Compared to Utilitarian Assignment

Coordinated MechanismStudent-‐Optimal

MatchingOrdinal Pareto Efficient

Matching

School ChoiceNeighborhood Assignment

24/30


Uncoordinated vs. Coordinated

Unlike these counterfactuals, we cannot easily simulate theuncoordinated mechanism

I We do observe the submitted ranks

Key challenge: What information is encoded in rik in uncoordinatedmechanism?

We consider three assumptions

1) Unordered selection: Ranked schools are better than unranked, butordering not truthful

2) Truthful selection: Schools ranked truthfully3) Unselected: Ignore the information contained in old ranks

Next results are from unordered selectionI Broadly similar conclusions from other two rules, though gains under

unselected assumption smaller

25/30

Welfare Distance-equivalent utility

Figure 4. Student Welfare from Uncoordinated and Coordinated Mechanism Distribution of utility (measured in distance units) from assignment based estimates in column 3 of Table A1 with mean utility in 2003-‐04 normalized to 0. Top and bottom 1% are not shown in figure. Line fit from Gaussian kernel with bandwidth chosen to

minimize mean integrated squared error using STATA’s kdensity command. 26/30


0.00

5.00

10.00

15.00

20.00

25.00

30.00

Welfare Gain from Assignment vs. Increased Travel Distance Welfare Gain Increase in Distance

27/30


Figure 5. Change in Student Welfare by Propensity to be

Administratively Assigned in the Uncoordinated Mechanism

Probability of administrative assignment estimated from probit of administrative assignment indicator on student census tract dummies and all student characteristics in the demand model except for distance. If student lives in tract where either all students are administratively or no students are

administratively assigned, all students from those tracts are coded as administratively assigned. Each student is assigned to one of ten deciles of probability of administrative assignment based on these estimates. Differences across deciles in distance-‐equivalent utility including distance, distance-‐equivalent utility net of

distance, and distance are plotted, where preference estimates come from column 3 of Table 7, under the selection assumption in Table 9.

28/30


0.00

5.00

10.00

15.00

20.00

25.00

30.00

Weflare Gain: Assignment vs. Enrollment Assignment Enrollment

29/30

Conclusion

Wrapping up

1) Preference heterogeneity generates substantial role for school choice

I New coordinated mechanism represents large improvement relative toneighborhood assignment counterfactual

2) Within choice system, gain from coordinating assignment roughly halfof that from having choice

a) Positive average gains across most student demographic groups,boroughs, and baseline achievement categories

X Staten Island and low baseline math experience greatest gains; highbaseline/SHSAT taking and Manhattan experience smallest

b) Gains increase in likelihood of administrative assignment

Little gain for students placed in the main round despite congestionUncoordinated mechanism’s deficiencies reflected in inefficientadministrative process

3) Effect of further tinkering of coordinated mechanism’s algorithm likelymodest

30/30

atila abdulkadiro glu (duke) nikhil agarwal (mit and nber) · nikhil agarwal (mit and nber) parag...

Documents