impact of nclb adequate yearly progress on district accountability in colorado carolyn haug,...

Impact of NCLB Adequate Yearly Progress on District Accountability

in Colorado

Carolyn Haug, Measured Progress

Jonathan Dings, Boulder Valley School District

AERA Annual Meeting 4/12/04

Presentation Overview

Evaluation FrameworkAYP Impact in Boulder Valley School DistrictColorado Context of Multiple Accountability

SystemsToward an Improved Accountability System

AERA Annual Meeting 4/12/04

Accountability System Evaluation Framework (Baker & Linn, 2004)

builds staff capacity; affects resource allocations; supports high-quality instruction; promotes student equity access to education; minimizes corruption; affects teacher quality, recruitment, and

retention; and produces unanticipated outcomes.

AERA Annual Meeting 4/12/04

Consequences

Of AYP Test-Score Driven AccountabilityApart from Consequences of

Comprehensive Federal Title Funding Changes

Apart from Consequences of Colorado’s 3rd-10th Grade State Testing Program, which predates NCLB’s AYP

AERA Annual Meeting 4/12/04

Results of Incentives

Slight Increase in Attention to Student Groups in Improvement Planning, Testing All; District, Schools Already Engaged

Bookkeeping for Disaggregation and AYP Status Calculation (350 person-hours)

Morale(?)

AERA Annual Meeting 4/12/04

AYP-Triggered Funding Impact

Apart from Broader NCLB/Title Funding Changes, Professional Development, Parent Engagement, Homeless Services

Expected Net Decrease in Discretionary $ Further Professional DevelopmentDollars Reserved for Transportation, A

Doubtfully Effective Use Fewer Schools and Students Served

AERA Annual Meeting 4/12/04

District Groups Not Making AYP in 2003

% of Targets Missed

Overall 0

Asian 0

White 0

Black 0

American Indian 0

Hispanic 33

English Language Learners 83

Students With Disabilities 33

Free/Reduced Lunch* (not in AYP) 17

Effects of Publicizing AYP Results

Schools Fail Targets

Banner Headline from Boulder Daily Camera Newspaper, 11/19/03

AERA Annual Meeting 4/12/04

Impact of AYP Ratings on the Media, Public, and Parents

Schools Fail Targets; Excellent Failures; Excellence Fails to Impress Feds (Boulder Daily Camera articles)

AYP status created dissonance about previously-held beliefs about some BVSD schools

Statewide, similar confusion prevailed: “The great power of AYP is that it doesn’t let Colorado’s best schools cover up with overall good scores those students being left behind.” (emphasis added)

AERA Annual Meeting 4/12/04

AYP Competes with Other Colorado Accountability Systems: School Accountability Reports (SAR) and AccreditationAs a result of three separately-enacted laws,

Colorado schools are subject to three different school accountability mechanisms:

AYP: federal law, enacted January 2002 SAR: state school reform legislation, enacted July 2000 Accreditation: state school reform legislation, enacted

July 1998

Subsequently, schools face potentially three different school ratings

AERA Annual Meeting 4/12/04

Attributes of System AYP SAR

Performance Model Achievement: status & growth; non-cohort

Achievement: status & growth; non-cohort

Decision-making Model

Conjunctive model using up to potentially numerous, depending on school size, indicators per school

Single composite score computed

Important CSAP Performance Cut-Scores

Partially Proficient All CSAP cut-scores

Methodology Partially Proficient, Proficient, Advanced performance contributes to making AYP

Standardized weighted index awards differential "credit" for each CSAP performance level

Sub-groups of Students Held Accountable For

Gender, Racial/Ethnic, Economically Disadvantaged, Students with Disabilities, English Language Learners

No subgroup reporting or analyses

Non-test Data Considered

None for elementary and middle schools, graduation rate for high schools

None for rating determination; other data is reported, including staff characteristics, safety indicators and budget information

BVSD School Accreditation

Effectiveness: status & growth; cohortCompensatory model using a wide variety of student and school performance indicators

All CSAP cut-scores

Weighted index from SAR, not standardized

Gender, Racial/Ethnic, Economically Disadvantaged, Students with Disabilities, English Language Learners, Talented and GiftedSeveral elements, including parent & community satisfaction, evidence of a safe & civil learning environment, and professional development for staff.

Complementary or Contradictory Systems?

AERA Annual Meeting 4/12/04

SAR and Accreditation Ratings for Schools Failing AYP

 SAR Rating Based

on Spring 2003School Accreditation Status,

2002-2003 School Year

School A Average Fully Accredited

School B High Fully Accredited

School C High Fully Accredited

School D Low Academic Watch

School E High Fully Accredited

School F Average Fully Accredited

School G Excellent Fully Accredited

School H Low Fully Accredited

School I Low Fully Accredited

AERA Annual Meeting 4/12/04

Size Mattered: Most Large Schools Failed AYP

 SAR Rating Based

on Spring 2003School Accreditation Status,

2002-2003 School Year

School A Average Fully Accredited

School B High Fully Accredited

School C High Fully Accredited

School D Low Academic Watch

School E High Fully Accredited

School F Average Fully Accredited

School G Excellent Fully Accredited

School H Low Fully Accredited

School I Low Fully Accredited

AERA Annual Meeting 4/12/04

Conclusion: Toward an Improved Accountability SystemAccountability is good when it accurately

identifies schools and when consequences are reasonable

Results from 3 systems are not synthesized, which leads to serious confusion

Multiple systems yield multiple measures that could be combined to form one contextual, cohesive synopsis of school performance

AERA Annual Meeting 4/12/04

Next Steps

The next task is to design a useful, valid methodology for integrating data from AYP, SAR and accreditation that meets the intended purposes of each of the 3 systems:

providing schools with useful feedback about performance in order to improve the school, and

school accountability. Rather than solely a school-shopping device, a

school’s rating would provide information, and therefore opportunities for improvement.