Impact of NCLB Adequate Yearly Progress on District Accountability

in Colorado

Carolyn Haug, Measured Progress

Jonathan Dings, Boulder Valley School District

AERA Annual Meeting 4/12/04

Presentation Overview

Evaluation FrameworkAYP Impact in Boulder Valley School DistrictColorado Context of Multiple Accountability

SystemsToward an Improved Accountability System

AERA Annual Meeting 4/12/04

Accountability System Evaluation Framework (Baker & Linn, 2004)

builds staff capacity; affects resource allocations; supports high-quality instruction; promotes student equity access to education; minimizes corruption; affects teacher quality, recruitment, and

retention; and produces unanticipated outcomes.

AERA Annual Meeting 4/12/04

Consequences

Of AYP Test-Score Driven AccountabilityApart from Consequences of

Comprehensive Federal Title Funding Changes

Apart from Consequences of Colorado’s 3rd-10th Grade State Testing Program, which predates NCLB’s AYP

AERA Annual Meeting 4/12/04

Results of Incentives

Slight Increase in Attention to Student Groups in Improvement Planning, Testing All; District, Schools Already Engaged

Bookkeeping for Disaggregation and AYP Status Calculation (350 person-hours)

Morale(?)

AERA Annual Meeting 4/12/04

AYP-Triggered Funding Impact

Apart from Broader NCLB/Title Funding Changes, Professional Development, Parent Engagement, Homeless Services

Expected Net Decrease in Discretionary $ Further Professional DevelopmentDollars Reserved for Transportation, A

Doubtfully Effective Use Fewer Schools and Students Served

AERA Annual Meeting 4/12/04

District Groups Not Making AYP in 2003

% of Targets Missed

Overall 0

Asian 0

White 0

Black 0

American Indian 0

Hispanic 33

English Language Learners 83

Students With Disabilities 33

Free/Reduced Lunch* (not in AYP) 17

Effects of Publicizing AYP Results

Schools Fail Targets

Banner Headline from Boulder Daily Camera Newspaper, 11/19/03

AERA Annual Meeting 4/12/04

Impact of AYP Ratings on the Media, Public, and Parents

Schools Fail Targets; Excellent Failures; Excellence Fails to Impress Feds (Boulder Daily Camera articles)

AYP status created dissonance about previously-held beliefs about some BVSD schools

Statewide, similar confusion prevailed: “The great power of AYP is that it doesn’t let Colorado’s best schools cover up with overall good scores those students being left behind.” (emphasis added)

AERA Annual Meeting 4/12/04

AYP Competes with Other Colorado Accountability Systems: School Accountability Reports (SAR) and AccreditationAs a result of three separately-enacted laws,

Colorado schools are subject to three different school accountability mechanisms:

AYP: federal law, enacted January 2002 SAR: state school reform legislation, enacted July 2000 Accreditation: state school reform legislation, enacted

July 1998

Subsequently, schools face potentially three different school ratings

AERA Annual Meeting 4/12/04

Attributes of System AYP SAR

Performance Model Achievement: status & growth; non-cohort

Achievement: status & growth; non-cohort

Decision-making Model

Conjunctive model using up to potentially numerous, depending on school size, indicators per school

Single composite score computed

Important CSAP Performance Cut-Scores

Partially Proficient All CSAP cut-scores

Methodology Partially Proficient, Proficient, Advanced performance contributes to making AYP

Standardized weighted index awards differential "credit" for each CSAP performance level

Sub-groups of Students Held Accountable For

Gender, Racial/Ethnic, Economically Disadvantaged, Students with Disabilities, English Language Learners

No subgroup reporting or analyses

Non-test Data Considered

None for elementary and middle schools, graduation rate for high schools

None for rating determination; other data is reported, including staff characteristics, safety indicators and budget information

BVSD School Accreditation

Effectiveness: status & growth; cohortCompensatory model using a wide variety of student and school performance indicators

All CSAP cut-scores

Weighted index from SAR, not standardized

Gender, Racial/Ethnic, Economically Disadvantaged, Students with Disabilities, English Language Learners, Talented and GiftedSeveral elements, including parent & community satisfaction, evidence of a safe & civil learning environment, and professional development for staff.

Complementary or Contradictory Systems?

AERA Annual Meeting 4/12/04

SAR and Accreditation Ratings for Schools Failing AYP

 SAR Rating Based

on Spring 2003School Accreditation Status,

2002-2003 School Year

School A Average Fully Accredited

School B High Fully Accredited

School C High Fully Accredited

School D Low Academic Watch

School E High Fully Accredited

School F Average Fully Accredited

School G Excellent Fully Accredited

School H Low Fully Accredited

School I Low Fully Accredited

AERA Annual Meeting 4/12/04

Size Mattered: Most Large Schools Failed AYP

 SAR Rating Based

on Spring 2003School Accreditation Status,

2002-2003 School Year

School A Average Fully Accredited

School B High Fully Accredited

School C High Fully Accredited

School D Low Academic Watch

School E High Fully Accredited

School F Average Fully Accredited

School G Excellent Fully Accredited

School H Low Fully Accredited

School I Low Fully Accredited

AERA Annual Meeting 4/12/04

Conclusion: Toward an Improved Accountability SystemAccountability is good when it accurately

identifies schools and when consequences are reasonable

Results from 3 systems are not synthesized, which leads to serious confusion

Multiple systems yield multiple measures that could be combined to form one contextual, cohesive synopsis of school performance

AERA Annual Meeting 4/12/04

Next Steps

The next task is to design a useful, valid methodology for integrating data from AYP, SAR and accreditation that meets the intended purposes of each of the 3 systems:

providing schools with useful feedback about performance in order to improve the school, and

school accountability. Rather than solely a school-shopping device, a

school’s rating would provide information, and therefore opportunities for improvement.