from admiring to using and practices objectives...•understand components of a research-based...

©2012KarenA.BlaseandDeanL.Fixsen 1

From Admiring to Using Data to Improve Systems and Practices

Caryn Ward, Ph.D.

Dale Cusumano, Ph.D.

Angela Preston Ph.D.

Objectives

• Understand components of a research-based problem solving model;

• Understand differences and appropriate contexts within which to use problem solving or improvement cycle approaches

• Gain skills in applying improvement model processes including general problem solving approaches to all levels and layers of schools as systems.

Problem

Solution

Effective?Efficient?

Out of

Time


Quick Peek at Active Implementation Frameworks

Effective & Usable

Innovations

Stages

©Copyright Dean Fixsen and Karen Blase

DRIVERS

§ Whatcriticalsupportsareneededtomakethischange?Whatistheinfrastructure?

STAGES

§ Whatstepsleadtosuccessfulimplementation?TEAMS

§ Whotakesresponsibilityforandhelpsguidethechangeprocess?EFFECTIVE&USABLEINNOVATIONS

§ Whatexactlyarepeoplesayinganddoingthatmakesthingsbetterforchildrenandfamilies?

IMPROVEMENTCYCLES

§ Howcanwecreatemorehospitableenvironments,efficientlysolveproblemsandgetbetter?

Drivers

Teams

Cycles

Effective & Usable

Innovations

Stages DriversTeams

Cycles

Active Implementation Framework in a Formula for Success

Problem

Identification

Plan

Evaluation

Did it

work?

How will

we do this?Plan

Development

Why?

Problem

Analysis

Is there a

problem?

Horner, R. H., Newton, J. S., Todd, A. W., Algozzine, B. Algozzine,

K., Cusumano, D. L., & Preston, A. I. (2015). Team Initiated Problem Solving. Online at www.PBIS.org

Problem

Identification

Plan

Evaluation

Did it

work?

How will

we do this?Plan

Development

Why?

Problem

Analysis

Is there a

problem?


Welcome to Cherry Elementary SchoolMeet the Building

Implementation

TeamTeams What When

Who

Why Where

Problem Identification begins with precision

Welcome to Cherry Elementary SchoolMeet the Building

Implementation

Team

Is there a problem?

What is the problem?

Who is having the problem?

Why?


Possible Problem Precision StatementThemajorityofthirdgradestudents(who)arenotreadingfluentlyorcomprehendingwrittentext(what,when,where)because…..??????????(why).

Problem

Identification

Why?

Problem

Analysis

Analyze your Active Implementation Frameworks

Effective & Usable

Innovations

Stages


DRIVERS

§ Arecriticalsupportsthatareneededtomakethechangepresent?Whatistheinfrastructure?

STAGES

§ Weretherightstepsandactivitiestakenattherighttime?TEAMS

§ Aretherightpeopletakingtakingresponsibilityforguidingthechangeprocess?EFFECTIVE&USABLEINNOVATIONS

§ Areinnovationslearnable,teachable,doable,andreadilyassessableinpractice?IMPROVEMENTCYCLES

§ Arehospitableenvironmentsestablishedtosupportefficientproblemsolvingandbarrierbusting?

Drivers

Teams

Cycles

Usable Innovations: How often are…

Core Components Operationally Defined?

Fidelity Assessments Correlated with Outcomes?

18% 7%

1,200+ Outcome StudiesMoncher & Prinz, 1991;Gresham, et al., 1993; Dane & Schneider, 1998; Durlak & DuPre, 2008


Hexagon Tool Usable Innovations

Fidelity Assessment

System

Clear Description

Essential FunctionsUsable

Innovation

Operational Definitions

Clear philosophy,

values, and

principles

Inclusion and

exclusion criteria

Identification of what

must be present

(core components)

What is done in

practice

(Practice Profiles).

Did we do what we

said we would do?

Usable Innovation

Performance Assessments

Clear Description


Innovation


Teachable

LearnableDoable

Assessable in practice

Performance Assessments

Clear Description


Innovation


Teachable

LearnableDoable

Assessable in practice

Does Cherry Elementary have an Effective and Usable Innovation?

✅✅

✅

✅


Do we have a research-based usable innovation?

Areweimplementing

itwell?

Analyze your Active Implementation Frameworks

Effective & Usable

Innovations

Stages


DRIVERS

§ Arecriticalsupportsthatareneededtomakethechangepresent?Whatistheinfrastructure?

STAGES

§ Weretherightstepsandactivitiestakenattherighttime?TEAMS

§ Aretherightpeopletakingtakingresponsibilityforguidingthechangeprocess?EFFECTIVE&USABLEINNOVATIONS

§ Areinnovationslearnable,teachable,doable,andreadilyassessableinpractice?IMPROVEMENTCYCLES

§ Arehospitableenvironmentsestablishedtosupportefficientproblemsolvingandbarrierbusting?

Drivers

Teams

Cycles

0

1

2

Clear Instruction Demonstrates Tasks Engages Students Provides Feedback Adjusts to Responses Provides Opportunities

Adjusts to Engagement

Total Average

Observation Tool for Instructional Supports and Systems (OTISS)Average Scores for 3rd Grade - Winter 2017

3rd

Fully in Place

Partiallyin Place

NotObserved

Adequate Quality Instruction

Clear Instruction

DemonstratesTasks

Provides Feedback

Provides Opportunities

Engages Students

Adjusts to Responses

Adjusts to Engagement


Meet the District Implementation Team (DIT)

Teams

What systems are in place to support the school?

Competency

Leadership Organization


Possible Problem Problem StatementThe majority of third grade students (who) are not reading fluently or

comprehending written text (what, when, where) because reading content has not been delivered with clear instruction, instructional tasks demonstrated and frequent opportunities to practice with feedback provided (why). It is hypothesized that this is due to limited coaching of teachers in the delivery of the reading innovation (why).

Problem

Identification

How will

we do this?Plan

Development

Why?

Problem

Analysis

Is there a

problem?

Goals across teams65% of 3rd grade students will meet

grade level expectations in reading

fluency by May of 2017

OTISS scores will increase by 0.5

overall by March

DCA Competency subscale will

increase to 65% by August of

2017

• Buildrefreshertraining

• Createcoachingplan

Plan Development- Competency Driver


Plan

Evaluation

How will

we do this?Plan

Development

Why?

Problem

Analysis

Is there a

problem?

Did it

work?

Problem

IdentificationAnalyze Fidelity and Outcome Data

OTISS scores will increase by 0.5

overall by March

65% of 3rd grade students will meet

grade level expectations in reading

fluency by May of 2017


Problem

Identification

Plan

Evaluation

Plan

Development

Problem

Analysis

Celebrate!

“Insanityisdoingthesamethingover

andoverandexpectingdifferentresults”

AlbertEinstein

"Ifwekeepdoingwhat

wehavebeendoing,

we'llkeepgettingwhat

we'vealwaysgotten”

PaulBatalden

“Ifwewantbetteroutcomes,wemustchange

somethinginthesystem.Todothis,weneedto

understandoursystems.”

DonaldBerwick

“Everysystemisperfectlydesignedtoget

theresultsitgets”

Senge Deming

Three Types of Improvement Cycles

Rapid Cycle

Problem SolvingUsability Testing

Practice-Policy

Communication

Cycle

Rapid cycle, short

term, few people

Initially identifying

problems and

solutions

Address immediate and

pressing issues

Make incremental

improvements quickly

Rapid Cycle (PDSA) Problem SolvingRapidcycle(PDSA)problemsolving

Shewhart (1931); Deming (1986); Taylor et al. (2014)

Plan

DoStudy

Act

What do you PLAN to do?

Can you do it in

practice?Did you do it?

What happened?

What can be changed

and improved?


Outcomes from a Meta-analysis of Improvement Studies (N = 73)

Plan

DoStudy

Act

(Taylor et al. 2014)

100%

??%15%

14%

19%Three Types of Improvement Cycles

Rapid Cycle


Practice-Policy

Communication

Cycle

Medium cycle, medium-

term, more people

Testing the feasibility

of solutions

Developing administrative

supports

Usability Testing Aplannedseriesoftestsofaninnovationorofimplementationprocessesthattest thefeasibilityandimpactofanewwayofworkpriortorollingoutmorebroadly

Rubin (1994); Nielsen (2000); Akin et al. (2013)

Plan

DoStudy

Act Plan

DoStudy

Act Plan

DoStudy

Act

More is learned from 4 cycles with 5 participants each than from 1 pilot test with 20 participants

Three Types of Improvement Cycles

Rapid Cycle


Practice-Policy

Communication

Cycle How are supports and barriers

leveraged across the system?


ImplementationTeams:Competencies&Functions

EnsuringSupport

EngagingtheCommunity

CreatingEnablingContext

Effective Practices

Implementation Infrastructure

Improvement Cycles

System Change

Linking Team Structure

Building

ImplementationTeam

District

ImplementationTeam

Regional

ImplementationTeam

State/System

ImplementationTeam• EnsureSupport

• EngagetheCommunity

• CreateEnablingContext

Implementation Team

StateManagement

Team

TeachersInnovations

Students

Syst

em

Cha

nge

Adaptive Challenges• Duplication

• Fragmentation

• Hiring criteria

• Salaries

• Credentialing

• Licensing

• Time/ scheduling

• Union contracts

• RFP methods

• Federal/ State laws

“Ext

erna

l” S

yste

m C

hang

e Su

ppor

t

Prac

tice-

Polic

y C

omm

unic

atio

n Lo

opPolicy Enabled Practice

“The fault

cannot lie in

the part

responsible

for the

repair.”

Ashby (1956)

Practice-Policy Feedback Loop

School“BIT”

District“DIT”

Regional Entity“RIT”

Problem Solving PDSA Cycles

Systematic

Use data

Resolve problems

Study and address needs of

practice in context

Process initiated by a

“problem”

Setting event = discrepancy between expected and

observed

Initiated to study the impact of “change”

Setting event = purposeful effort to

study practices

Making things betterClosing gaps

Planned“Reactive”


ActivityAnswer these questions:

1. Which approach (e.g., general problem

solving, PDSA cycles, usability testing)

would you consider using and why?

2. What question(s) will you answer or

what is your precise problem statement

you will try to define?

3. What is your goal or outcome?

4. What steps will you follow?

5. What data are needed, from whom,

and what is the frequency?

Example

ActivityAnswer these questions:

1. Which approach (e.g., general problem

solving, PDSA cycles, usability testing)

would you consider using and why?

2. What question(s) will you answer or

what is your precise problem statement

you will try to define?

3. What is your goal or outcome?

4. What steps will you follow?

5. What data are needed, from whom,

and what is the frequency?

Objectives

• Understand components of a research-based problem solving model;

• Understand differences and appropriate contexts within which to use problem solving or improvement cycle approaches

• Gain skills in applying improvement model processes including general problem solving approaches to all levels and layers of schools as systems.


Frank Porter Graham Child Development InstituteUniversity of North Carolina

Chapel Hill, NC

http://nirn.fpg.unc.edu/www.scalingup.org

www.implementationconference.org

Caryn Ward, Ph.D.Dale Cusumano, Ph.D.Angela Preston, Ph.D.

For More Information

Citation and CopyrightThis document is based on the work of the National Implementation Research

Network (NIRN).

© 2013-2016 Allison Metz, Leah Bartley, Jonathan Green, Laura Louison, Sandy

Naoom, Barbara Sims, and Caryn Ward

This content is licensed under Creative Commons license CC BY-NC-ND,

Attribution-NonCommercial-NoDerivs . You are free to share, copy, distribute and

transmit the work under the following conditions: Attribution — You must attribute

the work in the manner specified by the author or licensor (but not in any way that

suggests that they endorse you or your use of the work); Noncommercial — You

may not use this work for commercial purposes; No Derivative Works — You may

not alter, transform, or build upon this work. Any of the above conditions can be

waived if you get permission from the copyright holder.

email: [email protected]

web: http://nirn.fpg.unc.edu

The mission of the National Implementation Research Network (NIRN) is to

contribute to the best practices and science of implementation, organization

change, and system reinvention to improve outcomes across the spectrum of

human services.

from admiring to using and practices objectives...•understand components of a research-based...

Documents