Value-added Accountability for Value-added Accountability for Achievement in Minneapolis Achievement in Minneapolis Schools and ClassroomsSchools and Classrooms

Minneapolis Public SchoolsMinneapolis Public Schools

December, 2000December, 2000

www.mpls.k12.mn.us/REA/www.mpls.k12.mn.us/REA/

Minneapolis Pubic SchoolsMinneapolis Pubic SchoolsMeasuring Up Against High StandardsMeasuring Up Against High Standards MPS developed standards-based reading and math

tests - Northwest Achievement Levels Tests (NALT). All students participate Sensitive to growth Predictive of high stakes outcomes

Developed indicators in concert with stakeholders who are internal and external to the district.

Indicators of school performance are based on multiple indicators. 33 for elementary and middle schools 24 for high schools

Standards Based means:Standards Based means: We use a mutually agreed upon rubric for We use a mutually agreed upon rubric for

each indicator of school performance.each indicator of school performance. Rubric scoring on a “1” to “5” scale.Rubric scoring on a “1” to “5” scale.

““3” Current standard3” Current standard ““5” Long term goal5” Long term goal

Data are reported by site, by indicator.Data are reported by site, by indicator. Community reports list percent of indicators Community reports list percent of indicators

at standard (3 or better) “Measuring Up.”at standard (3 or better) “Measuring Up.”

Selecting a unit of AnalysisSelecting a unit of Analysis

Minneapolis Public Schools in concert Minneapolis Public Schools in concert with multiple stakeholders agreed upon with multiple stakeholders agreed upon a matrix of indicators.a matrix of indicators. School PersonnelSchool Personnel Community input (e.g. NAACP)Community input (e.g. NAACP) Chamber of CommerceChamber of Commerce Minneapolis FoundationMinneapolis Foundation FamiliesFamilies

Criteria for IndicatorsCriteria for Indicators

Meyer (1996)Meyer (1996) Indicators must measure things that matter Indicators must measure things that matter

or are valued by society (MCA, MBST).or are valued by society (MCA, MBST). Indicators must be able to avoid being Indicators must be able to avoid being

“corrupted.”“corrupted.” inclusion v. exclusion criteriainclusion v. exclusion criteria

Indicators must be able to measure the Indicators must be able to measure the unique contribution schools “add” for each unique contribution schools “add” for each measured outcome.measured outcome.

Stakeholders agreed on Stakeholders agreed on using…using… Multiple indicators of school Multiple indicators of school

performance.performance. See chart of indicators from Appendix.See chart of indicators from Appendix. These indicators include:These indicators include:

Current level of performanceCurrent level of performance cross-cohort cross-cohort Gain and value-added measuresGain and value-added measures

Elementary & Middle School Elementary & Middle School QPA IndicatorsQPA Indicators

Type Level Cross-Cohort Gain – ValueAdded

AchievementTests (24)

State Tests(5)

State Tests(5)

State Tests,Oral Reading,

NALT (14)

Correlatedwith

Achievement(9)

Attendance,Suspensions,

Safety,Respect,

Gifted Programs,Summer SchoolParticipation (7)

Attendance,Suspensions,

(2)

Fatal Flaws of a level or means Fatal Flaws of a level or means approach.approach.

Mean test performance is Mean test performance is contaminated contaminated by by factors other than school performance.factors other than school performance.

Mean performance is out of date, and does Mean performance is out of date, and does not address cumulative, compounding effects not address cumulative, compounding effects of previous instruction.of previous instruction.

Mean performance is affected by mobility.Mean performance is affected by mobility. Mean performance interferes with localizing Mean performance interferes with localizing

performance.performance.

Distribution of school average 1999 NALT Distribution of school average 1999 NALT mathematics scores by free or reduced price mathematics scores by free or reduced price

lunch percentageslunch percentages

R2 = 0.7901

30

40

50

60

70

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Percent free or reduced price lunch

NA

LT

Mat

h A

vera

ge N

CE

School P = 44.4 nce

Distribution of school average mathematics gain Distribution of school average mathematics gain scores on the 1999 NALT by free or reduced price scores on the 1999 NALT by free or reduced price

lunch percentages.lunch percentages.

R2 = 0.0074

-3-2-10123456789

10

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Percent free or reduced price lunch

NA

LT

Mat

h N

CE

Gai

n

School P = 6.3 nce gain

Cross-Cohort AnalysisCross-Cohort AnalysisDistrict LevelDistrict Level

Percent of MPS Students Passing Minnesota Mathematics Basic Standard Graduation Test in 8th Grade

28%21%

38%

22%

40%

55%

29%

72%74%

27%

45%

19%

33%

74%

24%

0%

20%

40%

60%

80%

100%

AmericanIndian

AfricanAmerican

AsianAmerican

Hispanic White

1998

1999

2000

Standard

The problem with cross-cohort The problem with cross-cohort analysis at the school levelanalysis at the school level

1200

1240

1280

1320

1360

1400

1440

1998 1999 2000 2001 2002 2003

Mean scale score

Grade 3 Reading

Grade 3 Math

Grade 5 Reading

Grade 5 Math

Expected Growth= 26 points/yearin 3rd grade reading

Look at the “bounce” incross-cohort analysis

Northwest Achievement Levels Test Validity Northwest Achievement Levels Test Validity Coefficients with Minnesota State TestsCoefficients with Minnesota State Tests

Grade/Measure Grade 3ReadingMCA

Grade 3MathMCA

Grade 5ReadingMCA

Grade 5MathMCA

Grade 8ReadingMBST

Grade 8 MathMBST

Grade 2 NALTa (n/a) .80(n= 3219)

Grade 3 NALTb .87(n=3785)

.87(n=3640)

Grade 4 NALTa .85(n= 3153)

.86(n=3097)

Grade 5 NALTb .88(n=3533)

.89(n=3484)

Grade 7 NALTc .82(n= 2608)

.88(n= 2631)

aNALT April, 1998 correlation with MCA March, 1999 for same subject area (predictive validity)bNALT April, 1999 correlation with MCA March, 1999 for same subject area (concurrent validity)cNALT April, 1998 correlation with MBST February, 1999 for same subject area (predictive validity)

Linkage with State StandardsLinkage with State Standards

NALT Scale Score Which Predicts Passage of theMinnesota Basic Standards Test (MBST) in 8th Grade

Grade Reading Scale Score Reading Scale Score

3 190 1994 197 2065 204 2146 209 2187 213 2258 218 232

True gain model: Growth True gain model: Growth CurvesCurves

Class of 2005 Longitudinal Reading Gains

190

200

210

220

230

Grade 3

Grade 4

Grade 5

Grade 6

Grade 7

Grade 8

Grade 9

Grade 10

Grade 11

Grade 12

NA

LT

Sc

ale

Sc

ore

(R

as

ch

Mo

de

l)

High Standard

Basic Standard

Class of 2005 (n=2006)

True Gain model: Percent of True Gain model: Percent of Students Making One Year GainsStudents Making One Year Gains

Percent of Students Making One Year Growth or More on NALT Reading and Math by Program

55% 53%47%

56%53% 51%47%

54%

0%

20%

40%

60%

80%

100%

All ELL SpecialEducation

Mainstream

Per

cen

tage

of

Stu

den

ts

Reading

Math

District Standard

1998 1999Top 1/5 73.70 77.00 226 StudentsMedian 52.10 57.25Bottom 1/5 31.50 39.80

1998-99 EmersonMath Concepts 20/20 Analysis

10

20

30

40

50

60

70

80

90

1998 1999

Year

Norm

al C

urv

e E

quiv

ale

nt

(NC

E)

Top 1/5

Median

Bottom 1/531st percentile54th percentile

64th percentile

87th percentile

90th percentile

19th percentile

1998 1999Top 1/5 73.18 78.20 168 StudentsMedian 47.65 55.30Bottom 1/5 29.90 35.10

1998-99 EmersonReading Comprehension 20/20 Analysis

10

20

30

40

50

60

70

80

90

1998 1999Year

Norm

al

Curv

e E

quiv

ale

nt

(NC

E)

Top 1/5

Median

Bottom 1/5

24th percentile46th percentile

60th percentile

86th percentile

91st percentile

17th percentile

Value-added ModelValue-added Model Primary question is thisPrimary question is this, “, “How can an equitable How can an equitable

accountability system be put in place when there is a accountability system be put in place when there is a large degree of variability in the way student large degree of variability in the way student characteristics are distributed among our schoolscharacteristics are distributed among our schools?”?” English language learnersEnglish language learners Special education programsSpecial education programs Race/ethnicityRace/ethnicity PovertyPoverty

MPS relies on an empirical- and literature-based MPS relies on an empirical- and literature-based model to make predictions, not set expectations! model to make predictions, not set expectations!

Schools are acknowledged/rewarded for “Schools are acknowledged/rewarded for “Beating Beating the Oddsthe Odds.”.”

Value-added Teacher EffectsValue-added Teacher Effects

Post-test reading score=Post-test reading score= Pretest reading scorePretest reading score + Free or reduced price lunch+ Free or reduced price lunch + Racial/ethnic code+ Racial/ethnic code + Neighborhood poverty concentration+ Neighborhood poverty concentration + Lives with both parents code+ Lives with both parents code + Limited English Proficiency status+ Limited English Proficiency status + Special Education status+ Special Education status + Teacher effects+ Teacher effects

Teachers who “beat the odds” in Teachers who “beat the odds” in second grade reading reported: second grade reading reported:

more use of small group instructionmore use of small group instruction more development of word attack skillsmore development of word attack skills more individual student oral readingmore individual student oral reading more guidance during initial practicemore guidance during initial practice more explicit and direct phonics more explicit and direct phonics

instructioninstruction more use of systematic motivation more use of systematic motivation

strategiesstrategies