RE-EXAMINING THE ROLE OF PROFESSIONAL DEVELOPMENT AND TRAINING EVALUATION THROUGH AN IMPLEMENTATION SCIENCE LENS

MICHELLE GRAEF & ROBIN LEAKE

NHSTES June 15, 2011

Presentation Overview

Children’s Bureau’s Child Welfare Implementation Centers

What do we mean by “implementation”?

Overview of one Implementation Science framework (NIRN)

Implications of framework for training and training evaluation

Implementation Centers and National Resource Centers

Improve child welfare administration and practice Meet Federal performance standards and

implement effective programs Pursue sustainable and positive systems change Improve outcomes for children, youth & families

Funded by the Children’s Bureau in 2008 to provide States and Territories, Tribes, and Tribal Consortia individualized training and technical assistance (T/TA) to:

States and Tribes are sometimes without the resources necessary to implement comprehensive strategic plans

National Resource Centers have limited resources to provide intensive, long-term technical assistance

Implementation Centers provide in-depth and long-term technical assistance and peer networking opportunities to States and Tribes

Implementation Centers: Filling the Gap

Geographic Assignments

What do we mean by “implementation”?

The process of moving an idea from concept to reality

“a specified set of activities designed to put into practice an activity or program of known dimensions” (NIRN)

Involves both: project management (managing the tasks,

resources, budget, process –technical) change management (managing the people

– adaptive)

In other words…

Science of Implementation

There is a body of research on effective implementation, mostly from research on evidenced-based practices

Child welfare systems have been slow to explore the science, often using “paper implementation”

The Implementation Centers are using several models to guide their work: National Implementation Research Network (NIRN) Phases of Change (John Kotter) ADKAR (Awareness, Desire, Knowledge, Ability,

Reinforcement) - Prosci

Selection

Training

Coaching

Performance Assessment

Systems Intervention

LEADERSHIP

Decision Support Data

System

Integrated &

Compensatory

CO

MPE

TEN

CY

Drive

rs

ORG

AN

IZATIO

NAL

Drivers

Facilitative Administration

© Fixsen & Blase, 2008

National Implementation Research Network: Implementation Drivers

Stages of ImplementationCore Activities

Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network

Stages of ImplementationCore Activities (cont’)

Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network

So does any of this really happen in child welfare agencies? ICs have developed a common measure

of our implementation process, based on NIRN drivers but includes others

Collecting data across all projects: summarize implementation drivers that

are salient, describe implementation activities

happening over life of projects, across multiple stages.

Intervention vs. Implementation Outcomes Important to differentiate between

intervention-level activity/outcomes and implementation-level activity/outcomes (Proctor et al, 2010).

Examples of implementation outcomes: Acceptability (user satisfaction) Adoption/uptake (utilization or intent to try) Feasibility (suitability for use, practical) Fidelity (adherence to program model) Penetration (spread across settings) Sustainability (level of institutionalization)

Implications for Role of Training

What is the role of training in the NIRN framework? How is this different/not different from the way training is usually conceptualized by CW agencies?

Training alone, no matter how well done, does not lead to successful implementation

Fixsen, Naoom, Blase, Friedman, Wallace, 2005

Implications for Framework

How does overall professional development fit within the NIRN framework?

Why is training depicted separately from coaching in the triangle? Is there a better way to model this?

Implications for Role of Training Evaluation

What is the role of training evaluation in this framework?

What are the boundaries between training evaluation and implementation evaluation? Is there a distinction? Should there be?

What is the implication for the work that we do?

Learn More

Implementation Centers Website

Northeast & Caribbean Implementation Center (NCIC)

http://ncic.muskie.usm.maine.edu

Atlantic Coast Child Welfare Implementation Center (ACCWIC)

www.accwic.org

Midwest Child Welfare Implementation Center (MCWIC)

www.mcwic.org

Mountains & Plains Child Welfare Implementation Center (MPCWIC)

www.mpcwic.org

Western & Pacific Child Welfare Implementation Center (WPIC)

www.wpicenter.org