telling the whole story alternative accountability in portland, oregon prepared for the alternative...

Telling the Whole StoryAlternative Accountability in Portland, Oregon

Prepared for the Alternative Accountability Policy Forum

November 2015

About usChristopher Mazzeo, Ph.D.Director of Evidence Use and Policy Education NorthwestDirector, REL Northwest

Carla Gay, MSW Vice Principal, Reconnection ServicesMultiple Pathways to GraduationPortland Public Schools

Matt EideSenior AdvisorEducation Northwest

Goal

Share Portland’s experience, and lessons learned on the development of multiple

quantitative and qualitative accountability

metrics for alternative schools.

Guiding Question

How do you determine what quality looks like in an alternative program?

Discuss this question with a neighbor

Just enough context…

The District Perspective

Nagging question…How do we get accurate data to

represent the work of our alternative schools and programs?

But it was all below the line

The District Perspective

Confluence of Events in 2012 brought the nagging question above the line. . .

National Youth Taskforce

Oregonian Article Budget Cuts

The District Perspective

With the confluence of events, developing an accountability

framework for alternative education options became an above the line

priority…

The Alternative Schools’ Perspective

Will the metrics tell the whole story?Can traditional metrics accurately determine quality programming and achievement in alternative settings?

We don’t want your cookie cutter!Alternative programs in Portland are strong because of their diversity and specificity!

But we believe in quality too.

The REL Perspective

Help move the work forward

An issue of equity An

important issue in our

region

What we didand what we learned…

The Portland Approach

Quantitative

Site VisitsQualitative

How did we do this?

JFF pointed us in the direction of Denver and Chicago school districtsDenver helped us map our process for a district wide segmentation analysisChicago shared their accountability templateWe convened a group of stakeholders including policy, research and providersWe convinced REL Northwest to facilitate. . .

Your turn…

Use your dots to indicate which three metrics would be best to determine what quality looks like in an alternative program.

The Portland Framework

Put it here

METRIC DESCRIPTION

Academic ProgressSKILL GROWTH Percent of students who meet or exceed growth targets in

Reading and Math on either MAP or CASAS

CREDIT ATTAINMENT Percent of students who meet targets for the number of credits earned for length of enrollment or earn their maximum required credits while in school

Successful CompletionONE YEAR HS GRADUATION RATE Percent of students eligible for graduation who graduate within

one yearCOLLEGE READY GED ATTAINMENT RATE

Percent of students who meet/exceed GED target composite score

POSTSECONDARY READINESS Percent of students who meet/exceed target college readiness scores on COMPASS or ACT

School ConnectionAVERAGE DAILY ATTENDANCE Percent of days attended by students enrolled at the school

GROWTH IN ATTENDANCE Percent of students that show growth in their individual daily attendance rates compared to their individual daily attendance rate in the previous school year

ANNUAL RETENTION RATE Percent of students enrolled at an alternative school and retained from the point of enrollment to the end of the year

SCHOOL CLIMATE TBD

The Report Card

What we learnedHave a third party facilitate the process and analyze the data

Make sure to attend to technical and leadership considerations

Collaborative process

Include state policymakers in the process

Programs need access to real-time data

The Dashboard

What’s missing?

Are these metrics sufficient to reliably determine alternative program quality?

Discuss with your neighbor

Telling more of the story

through qualitative data

Qualitative ProcessThe Process

Seven step iterative process:

– needs sensing, instrument creation, instrument feedback collection, instrument revision, pilot testing, data analysis and reporting.

Piloted focus groups and interviews with three alternative programs

– Administrators, teachers, students, and parents

Qualitative Process

The InstrumentFocus group and interview questions aligned with PPS successful schools framework

The ResultsEvaluation briefs highlighting program context

Qualitative Process

Lessons Learned

Challenging to schedule visits and interviews

Aligning protocol with PPS Successful Schools Framework was useful but limited findings

Student focus groups should include alumni and students who are struggling

Qualitative Process

Lessons Learned

The qualitative review process is labor intensive and time consuming

Data triangulation and member check

De-islandize alternative education

Include as many stakeholders as possible

Questions?

Where to reach us

Christopher MazzeoChristopher.Mazzeo@educationnorthwest.org

Carla Gaycgay@pps.net

Matt EideMatthew.Eide@educationnorthwest.org