Building Implementation Capacity to Improve Youth Outcomes

Allison Metz, Ph.D.Associate Director

National Implementation Research Network University of North Carolina at Chapel Hill

Agenda• The Challenge• Implementation Science• Plan for Change• Active Implementation Frameworks

The Challenge: Recognizing the Gaps• Science to Service Gap

– What is known to be effective is not what is selected to help students

• Implementation Gap– What is selected is not used with fidelity and good

outcomes– What is used with fidelity is not sustained for a useful

period of time– What is used with fidelity is not used on a scale

sufficient to broadly impact youth outcomes

Implementation Gap

RESEARCHPRACTICE

Implementation is defined as a specified set of activities designed to put into practice an activity or program of known dimensions.

IMPLEMENTATION

RESEARCH PRACTICEGAP

Why Focus on Implementation?

IMPLEMENTATION

“Children, youth, and families cannot benefit from interventions they do not experience.”

Effective NOT Effective

Effective

NOT Effective

IMPLEMENTATION

INT

ER

VE

NT

ION Actual Benefits

(Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999)

Inconsistent; Not Sustainable; Poor outcomes

Unpredictable or poor outcomes;

Poor outcomes; Sometimes harmful

from Mark Lipsey’s 2009 Meta-analytic overview of the primary factors that characterize effective juvenile offender interventions – “. . . in some analyses, the quality with which the intervention is implemented has been as strongly related to recidivism effects as the type of program, so much so that a well-implemented intervention of an inherently less efficacious type can outperform a more efficacious one that is poorly implemented.”

“Implementation science is the systematic study of variables and conditions that lead to full and effective use of evidence-based programs and other effective innovations in typical human service settings.”

—Blase and Fixsen, 2010National Implementation Research

Network

“Implementation Science”

Download at: http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/

Best data show these methods, when used alone Do not Result in Implementation as Intended– Diffusion/ Dissemination of information– Training – Passing laws/ mandates/ regulations– Providing funding/ incentives– Organization change/ reorganization

5 to 10% return on investmentNECESSARY BUT NOT SUFFICIENT

Plan for Change: “Making It Happen”Letting it happen

– Recipients are accountable

Helping it happen– Recipients are accountable

Making it happen– Purposeful use of implementation practice and science– Implementation teams are accountable

—Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou (2004);Fixsen, Blase, Duda, Naoom, & Van Dyke (2010)

To successfully implement and sustain evidence-based and evidence-informed youth violence prevention interventions, we need to know:

WHAT to doWhat is the intervention (e.g.identified prevention, intervention and enforcement strategies that are feasible and relevant)?

HOW to do itActive and effective implementation and sustainability frameworks (e.g. strategies to change and maintain behavior of adults)

WHO will do itOrganized, purposeful, & active implementation support from linked implementation teams

Q. How?

A. Effective Implementation• Changing the behavior of practitioners and administrators • Creating the setting conditions to facilitate these changes• Creating the processes to maintain and improve these

changes in both setting conditions and behavior of well-intentioned adults

• So that youth benefit

Active Implementation Frameworks

Implementation Drivers Implementation Stages

Improvement Cycles Implementation Teams

IMPLEMENTATION DRIVERS

Common features of successful supports to help make full and effective use of a wide variety of innovations

Core Implementation

Components

© Fixsen & Blase, 2008

Positive Outcomes for Youth

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers

LeadershipLeadership

Effective Youth Violence Prevention Practices

Capacity to provide direction and vision

Staff capacity to support youth with the selected practices

Institutional capacity to support practitioners in implementing practices with fidelity

Performance Assessment(Fidelity)

Coaching

Training

Selection

Systems Intervention

Facilitative Administration

Decision Support Data System

Adaptive

Technical

Integrated & Compensatory

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers

LeadershipLeadership

Improved youth outcomes

Interventions meet

Implementation

Consistent Use of

Prevention Innovations

© Fixsen & Blase, 2008

IMPLEMENTATION STAGES

Purposeful matching of critical implementation activities to the stage of the process

Stages AND Drivers

Implementation Takes

Time: 2 – 4 Years

EXPLORATIO

N

IN

STALL

ATION

INIT

IAL

IMPLE

MENTA

TION

FULL

IMPLE

MENTA

TIONDrivers

Drivers

Drivers

“DRIVERS”

Stages of Implementation

EXPLORATIONC

ompe

tenc

y D

river

s Organization D

rivers

Leadership Drivers

Integrated & Compensatory

—Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

“Pay now or Pay later”

Goals of Exploration

• Create readiness for change• Changing hearts and minds

• Examine degree to which the proposed strategies and practices meet the needs of our community and our youth

• Determine whether the strategies, practices, and implementation are desirable and feasible

The HexagonAn EBP Exploration Tool

NEED

FIT

RESOURCES

EVIDENCE

CAPACITY

READINESS

Fit with current Initiatives• School, district , state priorities• Organizational structures

Community values

Need in school, district, state• Academic & socially significant Issues• Parent & community perceptions of need• Data indicating need

Resources and supports for:• Curricula & Classroom• Technology supports (IT dept.)• Staffing• Training• Data Systems• Coaching & Supervision• Administration & system

Evidence• Outcomes – Is it worth it?• Fidelity data• Cost – effectiveness data• Number of studies• Population similarities• Diverse cultural groups• Efficacy or Effectiveness

Capacity to Implement• Staff meet minimum qualifications• Able to sustain Imp Drivers

• Financially • Structurally

• Buy-in process operationalized• Practitioners • Families

Readiness for Replication• Qualified purveyor• Expert or TA available• Mature sites to observe• Several replications• How well is it operationalized?• Are Imp Drivers operationalized?

The “Hexagon” can be used as a planning tool to evaluate evidence-based programs and practices during the Exploration Stage of Implementation.

Download available at:www.scalingup.org/tools-and-resources

EBP:

5 Point Rating Scale:High = 5; Medium = 3; Low = 1.Midpoints can be used and scored as a 2 or 4.

High Med Low

Need

Fit

Resource Availability

Evidence

Readiness for Replication

Capacity to Implement

Total Score

© National Implementation Research Network 2009-2012 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University

of Maryland

Stages of Implementation

Com

pete

ncy

Dri

vers

Organization D

rivers

Leadership Drivers

Integrated & Compensatory

INST

ALL

ATIO

N

—Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

“If you build it, they will come”. . .

but you actually have to build it!

Goals of Installation

• Structural and functional changes are made to support implementation

• Staff selection protocols developed

• First ‘practitioners’ selected

• Define and initiate training of first cohort of practitioners

• Develop coaching system and plans

• Evaluate readiness and sustainability of data systems (e.g. fidelity, outcomes)

Stages of Implementation

Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

EXPLORATION

INSTALLATIO

N

INITIAL

IMPLEMENTATION

“Get Started, then Get Better.”

Com

pete

ncy

Dri

vers

Organization D

riversLeadership Drivers

Integrated & Compensatory

Goals of Initial Implementation

• Work through the Awkwardness

• Provide training and coaching on the evidence-based practice, re-organization of school roles, functions and structures

• Make use of improvement cycles to resolve systems issues

• Continue buy-in efforts and manage expectations

• All the components of the program or innovation are at least partially in place and the implementation supports begin to function

Stages of Implementation

2 - 4

Years

Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

FULL

IMPLEMENTATION

“The only thing worse than failing and not

knowing why you failed, is succeeding and

not knowing why you succeeded.”

Goals of Full Implementation

• Maintaining and improving skills and activities throughout the system

• Components integrated, fully functioning

• Skillful practices by front line staff, supervisors, administrators (50% meet performance criteria)

• Changes in policy are reflected in practice at all levels

• Ready to be evaluated for expected outcomes

IMPROVEMENT CYCLES

Changing on purpose to support the new way of work

PDSA CyclesImprovement Cycles

Plan

DoStudy

Act

• New practices do not fare well in existing

organizational structures and systems

• Effective innovations are changed to fit the system, as opposed to existing systems changing to support effective innovations.

• People, organizations, and systems. . .• Cannot change everything at once (too big;

too complex; too many of them and too few of us)

• Cannot stop and re-tool (have to create the new in the midst of the existing)

• Cannot know what to do at every step (we will know it when we get there)

• Many outcomes are not predictable (who knew!?)

• Plan-Do-Study-Act Cycles Rapid cycle problem solving

(Shewhart; Deming) Transformation Zone Usability testing (Neilson; Rubin) Practice-policy communication loops

Practice-Policy Communication Cycle

Policy

Practice

Po

licy En

ables P

ractices

Plan

DoEx

tern

al I

mp

lem

enta

tio

n S

up

po

rt Policy

Practice

Structure

Procedure

Pra

ctic

e In

form

s P

oli

cy

Fee

db

ack

Stu

dy - A

ct

FORM SUPPORTS FUNCTION

IMPLEMENTATION TEAMS

Organized, expert assistance to develop and sustain an accountable and effective structure

Implementation Teams• Provide accountable and effective structure to move

intervention through stages of implementation • Scope of the initiative determines the number of teams

and the linked communication protocols needed• Focus is on

– Ongoing “buy-in” and readiness– Installing and sustaining the Implementation Drivers – Fidelity & Outcomes– Systems Alignment and Stage-based work– Problem-solving and sustainability

“We tend to focus on snapshots of isolated parts of the system and wonder why our deepest problems never seem to get solved.”

—Senge, 1990

Linked Team StructuresImplementation Teams

Site-basedImplementation Team

Site-basedImplementation Team

Community-basedImplementation Team

Community-basedImplementation Team

Regionally-basedImplementation Team

State-basedImplementation Team

State-basedImplementation Team

Who Makes Change Happen?

Implementation Teams

• INDICATORS CORE COMPETENCIES

• Representative• Accountable

Team Structure

• Formal/Practice Knowledge• Model fluency • Fully operationalize Intervention

Know the Intervention

• Best practices for Implementation Core components

• Stage-appropriate Work

Know Implementation

• Communicate Change• Use of data for decision making,

problem solving and feedback loops

Know Improvement Cycles

•Knowledge and skills for system building and components

Know Systems Change

SO THAT...

…can effect change throughout the system and

make decisions

…can promote implementation of core components,

adaptations, and infrastructure

…can guide implementation and build capacity throughout the organization and system

…can develop and follow-through on action planning

…can support efforts to improve access, reach or scale, improve connection, influence decision-

making

Examples of SuccessImplementation results (higher fidelity) and intervention results (improved outcomes) improve when Active Implementation Frameworks are used purposefully and systematically• Catawba County Child Wellbeing Project- 5 year period to develop, implement

and evaluate post-care service system for children exiting foster care; high fidelity and early indicators of success

• Colorado EPIC Project- Scale-up of MI using Active Implementation Frameworks in corrections

• EPIS Center at Penn State University - 419 age-grade cohorts over a 5-year period: youth in CTC communities using EBPs had significantly lower rates of delinquency, greater resistance to negative peer influence, stronger school engagement and better academic achievement

• Evidence-Based Model Purveyors – NFP, MST, FFT use many components of active implementation frameworks to achieve sustainable outcomes

Summary: “Making it Happen” for youth Purposeful selection of an effective and feasible “What”

Conceptualize a change process so that effective interventions for children and families can become embedded and sustained in socially complex settings “stage-matched activities” to guide the process “implementation drivers” to build the infrastructure

Improvement processes are critical the work is never done because the environment is in motion

Invest in the development of organized, “expert” implementation support

Stay Connected!

nirn.fpg.unc.edu www.scalingup.org

www.implementationconference.org

Allison.metz@unc.edunirn@unc.edu