march, 2010 patricia chamberlain, phd strategies for constructing & scaling up evidence-based...

M A R C H , 2 0 1 0PAT R I C I A C H A M B E R L A I N , P H D

Strategies for Constructing & Scaling Up Evidence-Based Practices

The FocusHow are evidence-based practices constructed (what

goes in to them and why)?How can the child outcomes and factors that predict

(or drive) those outcomes be measured within “real world” settings?

How can evidence-based practice models fit into existing public service systems like juvenile justice and child welfare?

How can evidence-based models be scaled up?

Create the Blueprint: Carefully Visualize and Define the Outcome

Specificity- “Arrests” and “days

incarcerated” versus “delinquency”

- Make it measurable (Observable, from multiple

sources, not only self- reports)

Parsimoniousy

Constructing an EBP:Develop the Plan

What do we want to make happen for whom?

Define specific desired outcomes & how they can be measured.

Resist the temptation to focus on too manyoutcomes. Keep the plan clean and focused.

Who is the focus?(exclusion/inclusion)

Primary Outcomes to Decrease

Primary Outcomes to Increase

Adolescent girls in JJS

-Crime-Drug use-Pregnancy

-School+ Peer relations

Children in foster care

-Placement disruptions-Behavior problems

-Stability-Reunification

Foster parents -Dropping out of providing care-Stress

-Parenting Skills-Support

What Goes into the Plan? We look for high quality studies that identify risk and protective factors that predict or have strong associations with the outcomes of interest

Randomized controlled trials are the strongest for inferring causality Longitudinal studies that examine development over the lifespan are

helpful because they provide information on when to intervene (developmental sensitivity)

Multiple studies constitute a strong evidence base

Which of the risk and protective factors found in the studies are potentially malleable (by us/you)?

Structural Plan

Sample Outcomes

< Criminal Offending< Drug Use< Pregnancy

> Positive Peer Relations> School Attendance

Sample Outcomes



MalleableProtective

Factor 2

MalleableProtective

Factor 2

MalleableRisk Factor

1


1


2


2

MalleableProtective

Factor 1

MalleableProtective

Factor 1

Engineering the Intervention

Outcomes



Outcomes



Protective Factor

2

Protective Factor

2

Risk Factor

1

Risk Factor

1

Risk Factor

2

Risk Factor

2

Protective Factor

1

Protective Factor

1

Intervention Components

Aimed at decreasing Risk Factor #1


Aimed increasing Protective Factor #1

Aimed at increasing Protective Factor #2






Testing the Impact of the Intervention

Outcomes



Outcomes



Protective Factor 1

Protective Factor 1











Protective Factor 2

Protective Factor 2

Absence of specific intervention components

Risk Factor 1

Risk Factor 2

Significantly less change in outcomes

Mediators

Logic Model for Intervention Effects

Outcomes



Outcomes



Intervention ComponentsIntervention Components

Existing Factors Expected to

Moderate Outcomes

-Number of CWS placements-Age at first placement-Number of changes in caregivers-Number of previous arrests

AgeGender

Existing Factors Expected to

Moderate Outcomes

-Number of CWS placements-Age at first placement-Number of changes in caregivers-Number of previous arrests

AgeGender

Foster parent training and support to implement tracking of specific behavior

Reinforcement of being where you are supposed to be

Daily point and level system

School Card

Supervision

Delinquent Peer Groups

Positive Adult Mentoring

Reinforcement for + school & home behaviors

p < .05

MTFC or Group Care

MTFC or Group Care

Fitting Research in to “Real World” Settings

Ask if outcomes being addressed and measures of outcomes are:

Feasible (do not increase burden)

Meaningful (fit their agenda)

Capitalize on their existing system data

Example in Child Welfare

Placement disruptions: Between 1/3 to 1/2 of children disrupt within the first 12 months of care.

Feasible- already tracked in CFSRs Meaningful- rates are high and desirable to decrease Capitalizes on their data and easy to count Costs increase exponentially as the # of disruptions increase

1 2 3 4 5 6 7+

90

80

70

60

50

40

30

20

10

0

%

Number of placements

Percent of placement failures based on number of prior placements

1 2 3 4 5 6 7+

90

80

70

60

50

40

30

20

10

0

%



1 2 3 4 5 6 7+

90

80

70

60

50

40

30

20

10

0

%



Using System Data to Predict Risk Level

Research-based Risk & Protective Factors for Disruption

Risk Factors Child behavioral problems Foster parent stress

Protective Factors Foster Parent support Behaviorally based parenting skills

Example of a measure of risk factor: Parent Daily Report A daily snapshot of risk and protective factors

0

5

10

15

20

25

30

35

7/20

/00

7/27

/00

8/3/

00

8/10

/00

8/17

/00

8/24

/00

8/31

/00

9/7/

00

9/14

/00

9/21

/00

9/28

/00

10/5

/00

10/1

2/00

10/1

9/00

10/2

6/00

11/2

/00

11/9

/00

11/1

6/00

11/2

3/00

11/3

0/00

12/7

/00

12/1

4/00

12/2

1/00

12/2

8/00

1/4/

01

1/11

/01

1/18

/01

Total PDR# of Beh.Linear (Total PDR)Linear (# of Beh.)

PDR Scores at Baseline Predict Placement Disruption A threshold effect: After 6 behaviors, every additional behavior on the

PDR increases the probability of disruption by 17%.

Baseline PDR

Fitt

ed L

og

Ha

zard

of

Pla

cem

en

t Dis

rupt

ion

0 5 10 15 20

01

23

Research on Uptake and Scaling Up Evidence-based Practices

In the US 90% of child serving agencies do no t use EBPsThe agencies that tend to innovate do so repeatedlyThe rich get richer and most fall behindThe Needs/Innovations paradox (the systems that are in most

need are least likely to innovate)

Scaling Up MTFC in California & Ohio

Who Where DisciplinePatti Chamberlain CR2P, Oregon PsychologyHendricks Brown U of Miami BiostatisticsLynne Marsenich Ca Institute for M.H. Social WorkTodd Sosna Ca Institute for M.H. PsychologyLarry Palinkas U of Southern CA AnthropologyLisa Saldana CR2P, Oregon PsychologyPeter Sprengelmeyer CR2P, Oregon PsychologyGerry Bouwman TFCC Inc, Oregon BusinessWei Wang U of South Florida BiostatisticsPatrick Kanary CIP, Ohio Social WorkCourtenay Padgett CR2P, Oregon Coordinator

Study Design

40 non-early adopting counties are randomized to: 2 implementation conditions (CDT or IND) 1 of 3 time frames (research resource issue: Cohorts #1, #2, #3) Quantitative and qualitative measures

- Assess stable non-malleable factors (population density, # of placements, % minority)

- Assess “dynamic” malleable factors expected to mediate implementation success (organizational factors, attitudes towards EBPs)- Clinical team factors (fidelity, competence, willingness)- Child and Family factors (behavior change, placement outcomes)

Implementation success/failureStages of Implementation Completion (SIC)

Design

Included / Excluded

Matched R. A. to Time & Condition

Included 1. No MTFC 2. Placed 6 or more (N = 40)

Excluded 1. Existing MTFC 2. Placed <6 (N = 19)

1. Population Size (urban / rural)

2. Percentage minority

3. Number placed

4. Poverty

Cohort 1: 2007

Cohort 2: 2008

Cohort 3: 2009

The Stages of Implementation Completion (SIC)Theoretical Premise

Includes steps that have been identified as essential to the successful adoption, implementation and sustainability of MTFC

Protocol is developed to measure the achievement of a model-adherent program aimed at obtaining outcomes similar to RCTs.

SIC stages are operationalized and sequential Engagement--the fit between community needs and the goals of MTFC Procuring fiscal resources Developing a feasible time-line Analyzing the impact of staff recruitment on the organization (readiness) Assessment of long-term sustainability

Stages of Implementation Completion (SIC) Measures Implementation @ Multiple Levels:

System, Practitioner, Child/Family

8 Stages: Who is Involved?1. Engagement System

2. Considering feasibility System

3. Planning/readiness System, Practitioner

4. Staff hired and trained Practitioner

5. Fidelity monitoring process in place Practitioner, Child/Family

6. Services and consultation begin Practitioner, Child/Family

7. Fidelity, competence, & adherence Practitioner, Child/Family

8. Sustainability (certification) System, Practitioner

Activities Within the 8 SIC Stages

Stage 1 Engagement 1.1 Date site is informed services/ program available 1.2 Date of interest indicated 1.3 Date agreed to consider implementation 1.4 Date declined to consider implementation; Stage 1 discontinued

Stage 3 Readiness planning 3.1 Date of cost / funding plan review 3.2 Date of staff sequence, timeline, hire plan review 3.3 Date of FP recruitment plan review 3.4 Date of referral criteria plan review 3.5 Date written implementation plan completed 3.6 Date Stage 3 discontinued

Stage 4 Staff hired & trained 4.1 Date Service Provider selected 4.2 Date 1st staff hired 4.3 Date clinical training scheduled 4.4 Date clinical training held Count of # of staff trained 4.5 Date FP training scheduled/held 4.6 Date Stage 4 discontinued

Stage 6 Services and Consultation to Services Begin 6.1 Date of first placement 6.2 Date of first consult call 6.3 Date of first clinical meeting video review (count of number of videos)

6.4 Date of first foster parent meeting video review (count of number of videos)

6.5 Date Stage 6 discontinued

Two Scales on the SIC

Quantity- performance date driven- tracks completion of activities

Quality- performance ratings driven- relies on ratings by sites & trainers

Example of Measuring Quantity (days)Stage 1

Time Variable Mean RangeTime to Decline 100.47 3-1020Time to Consent 70.75 0-533

A B C D E F G H I J K L M N O P Q R S T U V W X 1.1 X X X X X X X X X X X X X X X X X X X X X X X X 1.2 X X X X X X X X X X X X X X X X X X X X X X 1.3 1.4 X X 2.1 X X X X X X X X X X X X X X X X X X X X X 2.2 X X X X X X X X 2.3 X X X X X X X X 2.4 X X X X 2.5 X X 2.6 X X X X X X X X X X X X X 3.1 X X X X X X 3.2 X X X X X X X 3.3 X X X X X 3.4 X X X X X X 3.5 X X X X 3.6 X 4.1 X X X X X 4.2 X X X X X 4.3 X X X X X X 4.4 X X X X X 4.5 X X X X X 4.6 X 5.1 X X X X X 5.2 X X X X X 5.3 X X X X 5.4 X X X X X 5.5 6.1 X X X X X 6.2 X X X X X 6.3 X X X X X 6.4 X X X X X 6.5 7.1 X X X X 7.2 X X X X 7.3 X X 7.4 X 8.1 8.2 8.3 8.4

Black = Cohort 1, Blue = Cohort 2, Yellow = Cohort 3, Red = Discontinued, Beige Shading = Discontinue Activity

SIC Progress by County

Examples of Quality Measures

Stage 2: Consideration of Feasibility-Ratings of system leaders interest MTFC trainers-Stakeholders feedback System leaders-Stage 3: Planning and Readiness-Planning Meeting Impressions MTFC trainers-Ratings of helpfulness of planning activities Site participantsStage 4: Staff Hired and Trained-Pre-training Ratings of MTFC Clinical team-Trainer Impressions MTFC trainers-Trainee Impressions Clinical team-PS, FP, Team, Org Ratings MTFC trainers

Examples of Quality

How strongly do you subscribe to the MTFC model? Clinical

How much support for the MTFC program is there organizationally?

MTFC

Rate team member’s ability to engage well with others.

MTFC

Next Steps on the SIC

See sites through Stage 8Finalize most appropriate scale scoresAssess if implementation condition (CDT vs. IND) affects quantity and/or quality scalesAssess how quantity and quality are relatedUse of other study measures to validate the measure and assess its ability to predict successful implementationValidate with non-study MTFC sitesValidate with other EBPs

What it takes to Scale-Up Evidence-based Practices?

Top down and bottom-up buy in

Mapping the “fit” between the intervention and the mission of the agency/system

Assessing how the activities/structures of the intervention disrupt daily duties & requirements (paperwork, court appearances, home visits, on-call)

Plan for change and instability (leadership turnover, funding ends)

Early Results on Predictors of Implementation

Densely populated counties who placed the largest number of youth in placement were the fastest to consent

System leaders who had the largest social networks were the “fence sitters”

Systems with a positive organizational climate and high motivational readiness to change were the most likely to implement

References

- Chamberlain, P., Brown, C. H., Saldana, L., Reid, J., Wang, W., Marsenich, L., Sosna, T., Padgett, C., & Bouwman, G. (2008). Engaging and recruiting counties in an experiment on implementing evidence-based practice in California. Administration and Policy in Mental Health and Mental Health Research, 35(4), 250-260.

- Chamberlain, P., Saldana, L., Brown, H., & Leve, L. D. (in press). Implementation of multidimensional treatment foster care in California: A randomized control trial of an evidence-based practice. In M. Roberts-DeGennaro, & S. J. Fogel (Eds.), Empirically supported interventions for community and organizational change. Chicago: Lyceum.

- Hoagwood, K., & Olin, S. (2002). The NIMH blueprint for change report: Research priorities in child and adolescent mental health. Journal of American Academy of Child and Adolescent Psychiatry, 41, 760-767.

- NIMH (2004). Treatment research in mental illness: Improving the nation’s public mental health care through NIMH funded interventions research. Washington, DC: Author.

march, 2010 patricia chamberlain, phd strategies for constructing & scaling up evidence-based...

Documents

malleable risk factor

decreasing risk factor

protective factors

child outcomes

drug use pregnancy

specific desired outcomes

evidencebased practices

intervention components