evidence-based programs to benefit consumers in human … · the “evidence-based movement” is...

Dean L. Fixsen, Karen A. Blase,

Sandra F. Naoom, Frances WallaceNational Implementation Research Network

Louis de la Parte Florida Mental Health Insitute

Evidence-Based Programs to Benefit Consumers in Human Services

CMHO, Toronto 2006

Copyright © Dean L. Fixsen and Karen A. Blase, 2006

0

100

200

300

400

500

600

700

800

900

≤1972 1974 1976 1978 1980 1982

0

50

100

150

200

250

300

Cumulative

Couples

Cumulative

Homes

Fixsen, Blase, Timbers, & Wolf (2001)

Teaching–Family Replications


Evidence Based Movement

The “evidence-based movement”

is an international experiment to make better use of research findings in typical service settings.

The purpose is to produce greater benefits to consumers and society.


“Evidence”

Based Movement

Two or more high quality research studies using randomized group designs (within subject designs)

Preferably done by two or more independent research groups

Preferably summarized in meta- analyses of findings across studies


Evidence Based “Movement”

Diffusion of innovationsDissemination of informationScientist-practitioners and

practitioner-scientists will read, listen, understand, and make good use of the products of scienceConsumers everywhere will

benefit


We Have Resources

U. S. federal government spends over $95 billion a year on research to develop new treatments

SCIENCE


We Have Resources

U. S. federal government spends over $1.8 trillion a year on supports for services to people

SERVICE


Our Results: CurrentOver 30 websites displaying lists

of “evidence-based”

programs and practices

Over 600 evidence-based programs and practices that have been assessed and deemed to be “evidence-based”


Our Results: Current

Systems are fragmented and are characterized by highly variable, often ineffective, and sometimes harmful services to consumers

Reports from 2001 –

2006 by the Institute of Medicine, Surgeon General, President’s New Freedom Commission concerning health, mental health, substance abuse, racial/ethnic disparities, emergency services


Our Results: 25 YearsIn 1983, A Nation at Risk declared that American schools faced a “rising tide of mediocrity,”

and that

America was in danger of falling behind its international competitors because of the poor performance of its students.

National Commission on Excellence in Education (1983)

More than 20 years later, the National Assessment of Educational Progress showed the achievement of U.S. students is virtually identical to what it was in the early 1980s.

Grigg, Daane, Jin, & Campbell (2003)


Our Results: 25 Years"... In response to public and political

expressions of concern about the utilization of government-supported knowledge, a subcommittee on technology transfer and utilization has been established.”

(NIMH, 1977)

“Is the ‘bench to bedside’

transition becoming more effective?”

(U. S. Congress, 2003)


The Toronto DeclarationThe current evidence-based movement has failed.

Even with thousands of research studies, consumers of human services are no better off today than they were 25 years ago.

The evidence-based movement is dead

Long live the evidence-based movement



Any good experiment (even failed ones), gives us a chance to learn from the data

We might not like our results, but it is important to know what they are so we can make better decisions

What have we learned so far?



Focusing on methodological rigor to move science to service is not sufficient

Relying on passive/unplanned means to move science to service is not sufficient


Science to Service

SCIENCE SERVICEGAPIMPLEMENTATION


Implementation ReviewsHuman service prevention and treatment programs (e.g. substance abuse, adult / children’s MH, justice, health, education)

Advanced manufacturing technologies

Research-based clinical guidelines

Engineering: bridge maintenance

Hotel service management

National franchise operations

Cancer prevention & treatment


Ineffective MethodsExcellent experimental evidence for

what does not work

Diffusion/dissemination of information by itself does not

lead to successful

implementation (research literature, mailings, promulgation of practice guidelines)

Training alone, no matter how well done, does not

lead to successful

implementation


Ineffective MethodsExcellent evidence for what does

not work

Implementation by edict does not work

Implementation by “following the money”

does not work

Implementation without changing supporting roles and functions does not

workPaul Nutt (2002). Why Decisions Fail


What WorksEffective intervention

practices

+Effective implementation

practices

=

Good outcomes

for consumers


ImplementationAn evidence-based program is

one thing

Implementation of an evidence- based program is a very

different thing


What is missing?

99% = Investment in Intervention Research to develop solutions

1% = Investment in Implementation Research to make effective use of those solutions

(Up from ¼% in 1977)


EBPs & Implementation

From an implementation perspective, what do we need to know about evidence-based programs?


EBPs & ImplementationThe usability

of a program has little

to do with the quality or weight of the evidence regarding that program

Evidence on intervention effectiveness for specific populations helps us choose

what to

implement

Evidence on the effectiveness of the intervention does not

help implement the

program or practice successfully


EBPs & ImplementationCore intervention components

Clearly described (who/what)

Practical measure of fidelity

Fully operationalized (do/say)

Field tested (recursive revision)

Contextualized (org./systems fit)

Effective (worth the effort)


Implementation

What do we need to know about successful implementation methods?


Stages of Implementation

Implementation is not an event

A mission-oriented process involving multiple decisions, actions, and corrections


Stages of Implementation

Exploration

Installation

Initial Implementation

Full Implementation

Innovation

Sustainability

Implementation occurs in stages:

Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

Intervention Outcomes

0%

100%

Implementation Outcomes


INTEGRATED & COMPENSATORY

CONSULTATION & COACHING

CONSULTATION & COACHING

STAFF PERFORMANCE

EVALUATION

STAFF PERFORMANCE

EVALUATION

FACILITATIVE ADMINISTRATIVE

SUPPORTS

FACILITATIVE ADMINISTRATIVE

SUPPORTS

RECRUITMENT AND SELECTION RECRUITMENT

AND SELECTION

PRESERVICE TRAINING

PRESERVICE TRAINING

SYSTEMSINTERVENTIONS

SYSTEMSINTERVENTIONS

Implementation Drivers

DECISION SUPPORT DATA SYSTEMS

DECISION SUPPORT DATA SYSTEMS

OUTCOMES(% of Participants who Demonstrate Knowledge, Demonstrate

new Skills in a Training Setting, and Use new Skills in the Classroom)

TRAININGCOMPONENTS

Knowledge SkillDemonstration

Use in the Classroom

Theory and Discussion 10% 5% 0%

..+Demonstration in Training

30% 20% 0%

…+ Practice & Feedback in Training

60% 60% 5%

…+ Coaching in Classroom

95% 95% 95%

Joyce and Showers, 2002

Integrated & Compensatory


Integrated & Compensatory

Practitioners impact consumers

It is the job of directors, managers, and funders to align policies and structures to facilitate effective practitioner practices

There is no such thing as an “administrative decision”

–

they are

all treatment decisions


Who does the work?Purveyors

Implementation Teams

Intermediary Purveyor Organizations

Transformed provider organizations

Transformed human service systems


Active Purveyor RoleSuccessful implementation on a useful scale requires a purveyor

A group of individuals with expertise in an innovation who actively work to implement that innovation with fidelity and good effect

Purveyors accumulate data & experiential knowledge --

more

effective and efficient over time (information economics, K. Arrow)


Active Purveyor RoleChange the behavior of adult human service professionals

“Systems don’t change, people do”

(J.W.)

Change organizational structures, cultures, and climates

Change the thinking of system directors and policy makers

Successful and sustainable implementation of evidence-based practices and programs always requires organizational change.


Active Purveyor Role

OrganizationManagement (leadership, policy)

Administration (HR, structure)

Supervision (nature, content)

Practitioner

Federal, State, & Local ContextSystem of care

Purv

eyor

Gro

upSimultaneous, Multi-Level Interventions

Purveyor Activities(N = 577 items)

Implementation Stages

Explore Install Init Impl

Assessment 97% 1% 2%

Planning 20% 32% 48%

Training 3% 31% 66%

Coaching 8% 6% 86%

Evaluation 3% 23% 73%

Org Development 11% 16% 73%

System Intervention 37% 30% 33%


Implementation Capacity

None <10 per Yr ≈10 per Yr Dedicated Purveyor Group

50%

N ≈

600 Named EBP Developers

% Use of Core Implementation Components

Hypothetical Information

% ProgramDevelopers

University Based


Who pays for all this?Transformed human service

systems 30 years ago, information technology

was 0% of the budget and now is 5%

Cost-benefit ratios improve dramatically when benefit > zero

Investment in effective implementation results in long-term savings in operating costs


Organizational ChangePhillips, Baron, Blase, Fixsen,

and Maloney (1975)


Costs and SavingsImplementation Costs & Savings

(Inflation Adjusted)

60708090

100110120

1 Yr Pre During PostYear 1

PostYear 2

PostYear 3

Chan

ge in

Bud

get (

Perc

ent)


Budget Changes (4 years)

Pre Post

Administration 37.0% 16.6%

Living Units 56.0% 66.4%

Training & Eval. 1.6% 5.5%

Comm.-Based 5.4% 11.5%

Reduction in administrative staff, reduction in maintenance staff and repairs, reduction in food/staff costs, nearly 100% staff redeployment/rehiring into newly defined positions


A Sobering Observation

"All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get."

R. Spencer DarlingBusiness Expert

“Systems trump programs.Patrick McCarthy, Annie E. Casey Foundation


Implementation = Transformation

Core Intervention Components

Alignment Alignment


Effective Use of EBPs

GAPRESEARCH SERVICEIMPLEMENTATION

Drivers

Stages


Evidence-Based Movement

The evidence-based movement is dead

Long live the evidence-based movement


Thank YouWe thank the following for their support

Annie E. Casey Foundation (EBPs and cultural competence)

William T. Grant Foundation (implementation literature review)

Substance Abuse and Mental Health Services Administration

(implementation strategies grants; sub-contract for NREPP reviews; sub-contract for SOC analyses of implementation)Centers for Disease Control

(implementation research contract)National Institute of Mental Health

(research and training grants)Juvenile Justice and Delinquency Prevention

(program development and evaluation grants)


For More InformationDean L. Fixsen

813-974-4446

[email protected]

Karen A. Blase813-974-4463

[email protected]

National Implementation Research NetworkAt the Louis de la Parte Florida Mental Health Institute

University of South Florida

http://nirn.fmhi.usf.edu

mailto:[email protected]

mailto:[email protected]

http://nirn.fmhi.usf.edu/


For More InformationFixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

Download all or part of the monograph at:

http://nirn.fmhi.usf.edu/resources/publications/Mono graph/index.cfm

http://nirn.fmhi.usf.edu/resources/publications/Monograph/index.cfm

http://nirn.fmhi.usf.edu/resources/publications/Monograph/index.cfm

evidence-based programs to benefit consumers in human … · the “evidence-based movement” is...

Documents