Evaluating the Implementation of

Child Welfare System Change

Introduction• The evaluation of implementing system

change is complex.• The emerging science of implementation

provides multiple frameworks that guide the evaluation

Why Evaluate?

•Document what IS implemented

• Effectiveness of Intervention(s)

• Effectiveness of Implementation

• Inform both through the stages of implementation

Why Evaluate?• Development of ongoing Quality

Assurance framework and process for sustainability

• Impact that implementation has on child welfare system

Complexity of IC Evaluation

• Evaluation of implementation of change at multiple levels:–Practice (EBP and non EBPs)–Organization – System

Complexity of IC Evaluation

• We also evaluate:– The process of implementation– Levels of fidelity–Outcomes of the intervention(s)

Evaluation System Change Matrix

Logic Model • Each MCWIC implementation project

includes a logic model that:–Provides an overall framework for

implementation–Makes explicit the logical connections in a

complex project– Identifies products, outcomes, and system

impacts

Logic Model• Logic model development is key to

planning the evaluation• The logic model is explicitly referenced in

the development of:

– Evaluation plans outlining all project evaluation activities–Work plans providing a timeline for project

activities

Implementation Process Review• Links implementation drivers to the logic

model•Measures fidelity of the implementation

in reference to the logic model & work plan•Provides ongoing monitoring of

implementation • Identifies emerging Project TA needs

Implementation Driver

Driver Outputs

Outputs from Logic Model

Installation Activities During this Period

Discussion / Next Steps/Issues/Technical Asst.

SaliencyImportance of driver during this time period

Training Revision of existing DCF training to incorporate WICWA

2-day foundations training curriculum review began in February 2010. Teleconference between trainers, Intertribal TP Steering Committee & Green Bay TP has been held to discuss WICWA changes

ID method that can best be used to determine the training needs of individual counties. MCWIC TA in development of

Salience rating

LoMidHi

Evaluating Interventions & Implementation

•An implementation project often includes multiple specific interventions• Each specific intervention has specific

outcomes. • Each intervention contributes to the

implementation outcomes

Long Term Outcomes (examples)•Organizational structure & functions

identified to work more effectively with county service organizations (OH)• Improved intake information collected,

documented, & available (IN)•Child Welfare staff will have increased

understanding of the history and purpose of ICWA (WI)

System Impacts (examples)

• Improved working relationship between OFC and County Child Welfare Agencies (OH)• Improved customer satisfaction with

intake reporting experience (IN)•Agency culture reflects a strength based

perspective on parents (IA)

System Impacts / Child and Family Outcomes (examples)

• Increased number of cases maintaining tribal connection (WI)• Shorter length of stay in foster care (IA)• Increased use of relative care (IN)

Conclusions• Evaluation is not a scary addition to an

implementation project

Conclusions

•Rather, evaluation is an integral aspect of the implementation itself, providing:–A guiding logic to the implementation–A procedure for monitoring the progress of

the implementation–A means of assessing implementation

success as well as emerging needs and challenges

•Questions?

•Comments?

•Tomatoes?