miblsi schools’ implementation process and student outcomes anna l. harms michigan state...

MiBLSi Schools’ Implementation Process and Student Outcomes

Anna L. HarmsMichigan State University

MiBLSi State Conference 2009 1

Agenda

• Reasons for studying implementation and ways to do it

• Linking research to our schools’ data• Next steps• Questions and Feedback


The Status of Research

• Primary focus has been on developing and identifying practices. . .– National Reading Panel Reports– What Works Clearinghouse– Florida Center for Reading Research Reviews– OJJDP Model Programs– Center for the Study and Prevention of Violence Model Programs


What determines the evidence base for a practice?


• Independent randomized control trial is the gold standard

• Effect size (Cohen, 1988) :– Large: .80– Moderate: .50– Minimal/Weak: .20

Efficacy vs. Effectiveness(Christensen, Carlson, Valdez, 2003)

• Efficacy– controlled conditions– Conducted by innovation developers

• Effectiveness– External to the developers of an innovation– Replication– Under different conditions


RESEARCH PRACTICE

IMPLEMENTATION

Greenberg, Domitrovich, Graczyk, Zins (2005)


PLANNED INTERVENTION

PLANNED IMPLEMENTATIO

N SYSTEM

PROGRAM ASIMPLEMENTED

ACTUAL INTERVENTION

ACTUAL MPLEMENTATION

SUPPORT

ACTUAL MPLEMENTATION

SUPPORT

=

NIRN/SISEP

• Framework for Implementation• Stages of Implementation• Core Implementation Components• Multi-level Influences on Successful

Implementation


Effective Intervention Practices+Effective Implementation Strategies_______________________________= Positive Outcomes for Students

SISEP, 2009MiBLSi State Conference 2009 8

Getting into the Habit of Collecting, Analyzing, and Acting Upon Data


Problem Identification

Problem Analysis

Plan Selection

Plan Implementation

Plan Evaluation DATA &

DOCUMENTATION

Response to I________

• Intervention ?

• Instruction ?

• Implementation of evidence-based practices


Reasons for Studying and Monitoring Implementation

• Effort evaluation• Quality improvement• Documentation• Internal validity• Program theory• Process evaluation• Diffusion• Evaluation quality


Greenberg, M. T., Domitrovich, C. E., Graczyk, P. A., & Zins, J. E. (2005).

What tools can we use to measure implementation of

school-wide systems?


Tier 1 Implementation Tools

READING BEHAVIORPlanning and Evaluation Tool Effective Behavior Supports Team

Implementation Checklist

Effective Reading Supports Team Implementation Checklist

Effective Behavior Supports Self Assessment Survey

Observational Protocols School-wide Evaluation ToolPrinciple’s Reading Walkthrough

DocumentsBenchmarks of Quality

School Climate Survey


Tier 2 & 3 Implementation Tools

READING BEHAVIORIntervention Validity Checklists Checklist for Individual Student

SystemsIEP Implementation Validity Checks IEP Implementation Validity Checks


MiBLSi Mission Statement

“to develop support systems and sustained implementation of a data-driven, problem solving model in schools to help students become better readers with social skills necessary for success”


Our Data

COHORT START DATE SCHOOLS* YEARS OF DATAAVAILABLE

1 January 2003 15 4.5

2 February 2005 27 3.5

3 January 2006 50 2.5

4.1 January 2007 65 1.5

4.2 March 2007 27 1.3

4.3 June, 2007 11 1


* Refers to # of elementary schools included in this study.

• MiBLSi’s existing data• Elementary Schools (any combination of K-6)

Purpose of the Study

• To systematically examine schools’ process of implementing school-wide positive behavior supports and a school-wide reading model during participation with a statewide RtI project.

• To systematically examine the relation between implementation fidelity of an integrated three-tier model and student outcomes.


Conceptual Framework


PLANNED INTERVENTION

School-wide Positive Behavior Supports

Response to Intervention for

Reading

ACTUAL IMPLEMENTATION

Submission of Implementation

Checklists

Scores on Implementation

Checklists

STUDENT OUTCOMES

Office Discipline Referrals

Performance on Curriculum-Based Literacy Measures

Performance on State-Wide Standardized

Test in Reading

(Chen, 1998; Greenberg et al., 2005)

Measuring Implementation

• Effective Behavior Support Self Assessment Survey (EBS-SAS)• Spring of each school year• Total % implementation by building location

• Effective Behavior Support Team Implementation Checklist (EBS-TIC)

• 4 x per school year (quarterly)• Total % Implementation

• Planning and Evaluation Tool for Effective Reading Supports-Revised (PET-R)

• Fall of each school year• Total/Overall % implementation


THE PROCESSHOW LONG

SUSTAINABILITYASSOCIATED STUDENT OUTCOMES

BEHAVIOR + READING


Systems Implementation Research

• Expect 3-5 years for full implementation (Fixsen, Naoom, Blase, Friedman & Wallace, 2004; OSEP Center on Positive Behavioral Interventions and Supports, 2004; Sprague et al., 2001)

• Studies often split up implementation and outcomes (Reading First--U.S. Department of Education, 2006)

• View implementation at one point in time (McCurdy, Mannella & Eldridge, 2003); McIntosh, Chard, Boland & Horner, 2006; Mass-Galloway, Panyan, Smith & Wessendorf, 2008)

• A need for systematic research


THE PROCESSHOW LONG


BEHAVIOR + READING


Process and Progress

• Just as we measure student progress, we should also measure our progress toward implementation efforts.

• What is our current level of implementation?• What is our goal?• How do we get from here to there?


How do scores vary by year of implementation?


THE PROCESSHOW LONG


BEHAVIOR + READING


How long does it take?

2-5 years


At each year of implementation, what % of schools attain criterion

levels of implementation?



PET-R: COHORT 3 (N=50)

0-5 mo. 6-11 mo. 1:6-1:11 2:6-2:11 3:6-3:11 4:6-4:11

24(48%)

1(2%)

25 schools (50% did not attain criterion scores)


EBS-SAS: COHORT 3 (N=50)

0-5 mo.


6-11 mo. 1:0-1:5 2:0-2:5 3:0-3:5 4:0-4:5 5:0-5:5

13(26%)

2(4%)

14(28%)


EBS-TIC: COHORT 3 (N=50)

0-5 mo.


6-11 mo. 1:0-1:5 2:0-2:5 3:0-3:5 4:0-4:5 5:0-5:5

6(12%)

1(2%)

30(60%)

THE PROCESSHOW LONG


BEHAVIOR + READING


Sustainability

• Think and work –Up–Down–Out


What percent of schools that attain criterion levels of implementation are able to maintain or improve

their score in all subsequent years?



PET-R: COHORT 3 (N=50)

6-11 mo. 1:6-1:11

1(2%)

1


EBS-SAS: COHORT 3 (N=50)

0-5 mo. 6-11 mo. 1:0-1:5 2:0-2:5 3:0-3:5 4:0-4:5 5:0-5:5

13(26%)

2(4%)

14(28%)

2

12

2


EBS-TIC: COHORT 3 (N=50)

0-5 mo. 6-11 mo. 1:0-1:5 2:0-2:5 3:0-3:5 4:0-4:5 5:0-5:5

6(12%)

1(2%)

30(60%)

15

01

Another way of looking at implementation. . .


What % of implementation data do schools submit for each year of

implementation?


% of Schools Submitting PET-R Data Each Year

1 2 3 4 5 6C1 -- -- 93% 80% 73% 60%

C2 -- 78% 89% 78% -- --

C3 -- 90% 94% -- -- --

C4.1 -- 97% -- -- -- --

C4.2 -- 96% -- -- -- --

C4.3 91% -- -- -- -- --


% of Schools Submitting EBS-SAS Data Each Year

1 2 3 4 5 6C1 -- -- 60% 60% 47% 53%

C2 70% -- 74% 63% 67% --

C3 84% -- 70% 78% -- --

C4.1 95% -- 86% -- -- --

C4.2 89% -- 81% -- -- --

C4.3 -- 82% -- -- -- --


% of Schools Submitting EBS-TIC Data Each Year


1 2 3 4 5 6C1 -- -- 47% 53% 73% 53%

C2 74% -- 78% 70% 56% --

C3 60% -- 80% 58% -- --

C4.1 77% -- 80% -- -- --

C4.2 56% -- 48% -- -- --

C4.3 -- 45% -- -- -- --

THE PROCESSHOW LONG


BEHAVIOR + READING


Is the % of behavior checklist data submitted each year related to

student behavior outcomes for that year?


Is the % of reading checklist data submitted each year related to

student reading outcomes for that year?


Are scores on the behavior implementation checklists related to student behavior outcomes for that

year?


Are scores on the reading implementation checklist for each year of implementation related to student reading outcomes for that

year?


THE PROCESSHOW LONG


BEHAVIOR + READING


What is the impact on student outcomes when schools meet

criteria on none, some, or all of the implementation checklists?


Limitations

• Self-report implementation measures• Limited number of schools in earlier cohorts• We don’t know what specific factors have

impacted implementation


Remember. . .

•More data is not necessarily better.

• Data should have a purpose: – It should help us to make well-informed decisions

that will improve outcomes for students.


miblsi schools’ implementation process and student outcomes anna l. harms michigan state...

Documents

datamiblsi state conference

implementation of evidence

implementation of school

student outcomesanna

itlinking research

wide systems

evidence base

gold standardeffect