march, 2010 patricia chamberlain, phd strategies for constructing & scaling up evidence-based...
Post on 19-Dec-2015
213 Views
Preview:
TRANSCRIPT
M A R C H , 2 0 1 0PAT R I C I A C H A M B E R L A I N , P H D
Strategies for Constructing & Scaling Up Evidence-Based Practices
The FocusHow are evidence-based practices constructed (what
goes in to them and why)?How can the child outcomes and factors that predict
(or drive) those outcomes be measured within “real world” settings?
How can evidence-based practice models fit into existing public service systems like juvenile justice and child welfare?
How can evidence-based models be scaled up?
Create the Blueprint: Carefully Visualize and Define the Outcome
Specificity- “Arrests” and “days
incarcerated” versus “delinquency”
- Make it measurable (Observable, from multiple
sources, not only self- reports)
Parsimoniousy
Constructing an EBP:Develop the Plan
What do we want to make happen for whom?
Define specific desired outcomes & how they can be measured.
Resist the temptation to focus on too manyoutcomes. Keep the plan clean and focused.
Who is the focus?(exclusion/inclusion)
Primary Outcomes to Decrease
Primary Outcomes to Increase
Adolescent girls in JJS
-Crime-Drug use-Pregnancy
-School+ Peer relations
Children in foster care
-Placement disruptions-Behavior problems
-Stability-Reunification
Foster parents -Dropping out of providing care-Stress
-Parenting Skills-Support
What Goes into the Plan? We look for high quality studies that identify risk and protective factors that predict or have strong associations with the outcomes of interest
Randomized controlled trials are the strongest for inferring causality Longitudinal studies that examine development over the lifespan are
helpful because they provide information on when to intervene (developmental sensitivity)
Multiple studies constitute a strong evidence base
Which of the risk and protective factors found in the studies are potentially malleable (by us/you)?
Structural Plan
Sample Outcomes
< Criminal Offending< Drug Use< Pregnancy
> Positive Peer Relations> School Attendance
Sample Outcomes
< Criminal Offending< Drug Use< Pregnancy
> Positive Peer Relations> School Attendance
MalleableProtective
Factor 2
MalleableProtective
Factor 2
MalleableRisk Factor
1
MalleableRisk Factor
1
MalleableRisk Factor
2
MalleableRisk Factor
2
MalleableProtective
Factor 1
MalleableProtective
Factor 1
Engineering the Intervention
Outcomes
< Criminal Offending< Drug Use< Pregnancy
> Positive Peer Relations> School Attendance
Outcomes
< Criminal Offending< Drug Use< Pregnancy
> Positive Peer Relations> School Attendance
Protective Factor
2
Protective Factor
2
Risk Factor
1
Risk Factor
1
Risk Factor
2
Risk Factor
2
Protective Factor
1
Protective Factor
1
Intervention Components
Aimed at decreasing Risk Factor #1
Aimed at decreasing Risk Factor #2
Aimed increasing Protective Factor #1
Aimed at increasing Protective Factor #2
Intervention Components
Aimed at decreasing Risk Factor #1
Aimed at decreasing Risk Factor #2
Aimed increasing Protective Factor #1
Aimed at increasing Protective Factor #2
Testing the Impact of the Intervention
Outcomes
< Criminal Offending< Drug Use< Pregnancy
> Positive Peer Relations> School Attendance
Outcomes
< Criminal Offending< Drug Use< Pregnancy
> Positive Peer Relations> School Attendance
Protective Factor 1
Protective Factor 1
Intervention Components
Aimed at decreasing Risk Factor #1
Aimed at decreasing Risk Factor #2
Aimed increasing Protective Factor #1
Aimed at increasing Protective Factor #2
Intervention Components
Aimed at decreasing Risk Factor #1
Aimed at decreasing Risk Factor #2
Aimed increasing Protective Factor #1
Aimed at increasing Protective Factor #2
Protective Factor 2
Protective Factor 2
Absence of specific intervention components
Risk Factor 1
Risk Factor 2
Significantly less change in outcomes
Mediators
Logic Model for Intervention Effects
Outcomes
< Criminal Offending< Drug Use< Pregnancy
> Positive Peer Relations> School Attendance
Outcomes
< Criminal Offending< Drug Use< Pregnancy
> Positive Peer Relations> School Attendance
Intervention ComponentsIntervention Components
Existing Factors Expected to
Moderate Outcomes
-Number of CWS placements-Age at first placement-Number of changes in caregivers-Number of previous arrests
AgeGender
Existing Factors Expected to
Moderate Outcomes
-Number of CWS placements-Age at first placement-Number of changes in caregivers-Number of previous arrests
AgeGender
Foster parent training and support to implement tracking of specific behavior
Reinforcement of being where you are supposed to be
Daily point and level system
School Card
Supervision
Delinquent Peer Groups
Positive Adult Mentoring
Reinforcement for + school & home behaviors
Fitting Research in to “Real World” Settings
Ask if outcomes being addressed and measures of outcomes are:
Feasible (do not increase burden)
Meaningful (fit their agenda)
Capitalize on their existing system data
Example in Child Welfare
Placement disruptions: Between 1/3 to 1/2 of children disrupt within the first 12 months of care.
Feasible- already tracked in CFSRs Meaningful- rates are high and desirable to decrease Capitalizes on their data and easy to count Costs increase exponentially as the # of disruptions increase
1 2 3 4 5 6 7+
90
80
70
60
50
40
30
20
10
0
%
Number of placements
Percent of placement failures based on number of prior placements
1 2 3 4 5 6 7+
90
80
70
60
50
40
30
20
10
0
%
Number of placements
Percent of placement failures based on number of prior placements
1 2 3 4 5 6 7+
90
80
70
60
50
40
30
20
10
0
%
Number of placements
Percent of placement failures based on number of prior placements
Using System Data to Predict Risk Level
Research-based Risk & Protective Factors for Disruption
Risk Factors Child behavioral problems Foster parent stress
Protective Factors Foster Parent support Behaviorally based parenting skills
Example of a measure of risk factor: Parent Daily Report A daily snapshot of risk and protective factors
0
5
10
15
20
25
30
35
7/20
/00
7/27
/00
8/3/
00
8/10
/00
8/17
/00
8/24
/00
8/31
/00
9/7/
00
9/14
/00
9/21
/00
9/28
/00
10/5
/00
10/1
2/00
10/1
9/00
10/2
6/00
11/2
/00
11/9
/00
11/1
6/00
11/2
3/00
11/3
0/00
12/7
/00
12/1
4/00
12/2
1/00
12/2
8/00
1/4/
01
1/11
/01
1/18
/01
Total PDR# of Beh.Linear (Total PDR)Linear (# of Beh.)
PDR Scores at Baseline Predict Placement Disruption A threshold effect: After 6 behaviors, every additional behavior on the
PDR increases the probability of disruption by 17%.
Baseline PDR
Fitt
ed L
og
Ha
zard
of
Pla
cem
en
t Dis
rupt
ion
0 5 10 15 20
01
23
Research on Uptake and Scaling Up Evidence-based Practices
In the US 90% of child serving agencies do no t use EBPsThe agencies that tend to innovate do so repeatedlyThe rich get richer and most fall behindThe Needs/Innovations paradox (the systems that are in most
need are least likely to innovate)
Scaling Up MTFC in California & Ohio
Who Where DisciplinePatti Chamberlain CR2P, Oregon PsychologyHendricks Brown U of Miami BiostatisticsLynne Marsenich Ca Institute for M.H. Social WorkTodd Sosna Ca Institute for M.H. PsychologyLarry Palinkas U of Southern CA AnthropologyLisa Saldana CR2P, Oregon PsychologyPeter Sprengelmeyer CR2P, Oregon PsychologyGerry Bouwman TFCC Inc, Oregon BusinessWei Wang U of South Florida BiostatisticsPatrick Kanary CIP, Ohio Social WorkCourtenay Padgett CR2P, Oregon Coordinator
Study Design
40 non-early adopting counties are randomized to: 2 implementation conditions (CDT or IND) 1 of 3 time frames (research resource issue: Cohorts #1, #2, #3) Quantitative and qualitative measures
- Assess stable non-malleable factors (population density, # of placements, % minority)
- Assess “dynamic” malleable factors expected to mediate implementation success (organizational factors, attitudes towards EBPs)- Clinical team factors (fidelity, competence, willingness)- Child and Family factors (behavior change, placement outcomes)
Implementation success/failureStages of Implementation Completion (SIC)
Design
Included / Excluded
Matched R. A. to Time & Condition
Included 1. No MTFC 2. Placed 6 or more (N = 40)
Excluded 1. Existing MTFC 2. Placed <6 (N = 19)
1. Population Size (urban / rural)
2. Percentage minority
3. Number placed
4. Poverty
Cohort 1: 2007
Cohort 2: 2008
Cohort 3: 2009
The Stages of Implementation Completion (SIC)Theoretical Premise
Includes steps that have been identified as essential to the successful adoption, implementation and sustainability of MTFC
Protocol is developed to measure the achievement of a model-adherent program aimed at obtaining outcomes similar to RCTs.
SIC stages are operationalized and sequential Engagement--the fit between community needs and the goals of MTFC Procuring fiscal resources Developing a feasible time-line Analyzing the impact of staff recruitment on the organization (readiness) Assessment of long-term sustainability
Stages of Implementation Completion (SIC) Measures Implementation @ Multiple Levels:
System, Practitioner, Child/Family
8 Stages: Who is Involved?1. Engagement System
2. Considering feasibility System
3. Planning/readiness System, Practitioner
4. Staff hired and trained Practitioner
5. Fidelity monitoring process in place Practitioner, Child/Family
6. Services and consultation begin Practitioner, Child/Family
7. Fidelity, competence, & adherence Practitioner, Child/Family
8. Sustainability (certification) System, Practitioner
Activities Within the 8 SIC Stages
Stage 1 Engagement 1.1 Date site is informed services/ program available 1.2 Date of interest indicated 1.3 Date agreed to consider implementation 1.4 Date declined to consider implementation; Stage 1 discontinued
Stage 3 Readiness planning 3.1 Date of cost / funding plan review 3.2 Date of staff sequence, timeline, hire plan review 3.3 Date of FP recruitment plan review 3.4 Date of referral criteria plan review 3.5 Date written implementation plan completed 3.6 Date Stage 3 discontinued
Stage 4 Staff hired & trained 4.1 Date Service Provider selected 4.2 Date 1st staff hired 4.3 Date clinical training scheduled 4.4 Date clinical training held Count of # of staff trained 4.5 Date FP training scheduled/held 4.6 Date Stage 4 discontinued
Stage 6 Services and Consultation to Services Begin 6.1 Date of first placement 6.2 Date of first consult call 6.3 Date of first clinical meeting video review (count of number of videos)
6.4 Date of first foster parent meeting video review (count of number of videos)
6.5 Date Stage 6 discontinued
Two Scales on the SIC
Quantity- performance date driven- tracks completion of activities
Quality- performance ratings driven- relies on ratings by sites & trainers
Example of Measuring Quantity (days)Stage 1
Time Variable Mean RangeTime to Decline 100.47 3-1020Time to Consent 70.75 0-533
A B C D E F G H I J K L M N O P Q R S T U V W X 1.1 X X X X X X X X X X X X X X X X X X X X X X X X 1.2 X X X X X X X X X X X X X X X X X X X X X X 1.3 1.4 X X 2.1 X X X X X X X X X X X X X X X X X X X X X 2.2 X X X X X X X X 2.3 X X X X X X X X 2.4 X X X X 2.5 X X 2.6 X X X X X X X X X X X X X 3.1 X X X X X X 3.2 X X X X X X X 3.3 X X X X X 3.4 X X X X X X 3.5 X X X X 3.6 X 4.1 X X X X X 4.2 X X X X X 4.3 X X X X X X 4.4 X X X X X 4.5 X X X X X 4.6 X 5.1 X X X X X 5.2 X X X X X 5.3 X X X X 5.4 X X X X X 5.5 6.1 X X X X X 6.2 X X X X X 6.3 X X X X X 6.4 X X X X X 6.5 7.1 X X X X 7.2 X X X X 7.3 X X 7.4 X 8.1 8.2 8.3 8.4
Black = Cohort 1, Blue = Cohort 2, Yellow = Cohort 3, Red = Discontinued, Beige Shading = Discontinue Activity
SIC Progress by County
Examples of Quality Measures
Stage 2: Consideration of Feasibility-Ratings of system leaders interest MTFC trainers-Stakeholders feedback System leaders-Stage 3: Planning and Readiness-Planning Meeting Impressions MTFC trainers-Ratings of helpfulness of planning activities Site participantsStage 4: Staff Hired and Trained-Pre-training Ratings of MTFC Clinical team-Trainer Impressions MTFC trainers-Trainee Impressions Clinical team-PS, FP, Team, Org Ratings MTFC trainers
Examples of Quality
How strongly do you subscribe to the MTFC model? Clinical
How much support for the MTFC program is there organizationally?
MTFC
Rate team member’s ability to engage well with others.
MTFC
Next Steps on the SIC
See sites through Stage 8Finalize most appropriate scale scoresAssess if implementation condition (CDT vs. IND) affects quantity and/or quality scalesAssess how quantity and quality are relatedUse of other study measures to validate the measure and assess its ability to predict successful implementationValidate with non-study MTFC sitesValidate with other EBPs
What it takes to Scale-Up Evidence-based Practices?
Top down and bottom-up buy in
Mapping the “fit” between the intervention and the mission of the agency/system
Assessing how the activities/structures of the intervention disrupt daily duties & requirements (paperwork, court appearances, home visits, on-call)
Plan for change and instability (leadership turnover, funding ends)
Early Results on Predictors of Implementation
Densely populated counties who placed the largest number of youth in placement were the fastest to consent
System leaders who had the largest social networks were the “fence sitters”
Systems with a positive organizational climate and high motivational readiness to change were the most likely to implement
References
- Chamberlain, P., Brown, C. H., Saldana, L., Reid, J., Wang, W., Marsenich, L., Sosna, T., Padgett, C., & Bouwman, G. (2008). Engaging and recruiting counties in an experiment on implementing evidence-based practice in California. Administration and Policy in Mental Health and Mental Health Research, 35(4), 250-260.
- Chamberlain, P., Saldana, L., Brown, H., & Leve, L. D. (in press). Implementation of multidimensional treatment foster care in California: A randomized control trial of an evidence-based practice. In M. Roberts-DeGennaro, & S. J. Fogel (Eds.), Empirically supported interventions for community and organizational change. Chicago: Lyceum.
- Hoagwood, K., & Olin, S. (2002). The NIMH blueprint for change report: Research priorities in child and adolescent mental health. Journal of American Academy of Child and Adolescent Psychiatry, 41, 760-767.
- NIMH (2004). Treatment research in mental illness: Improving the nation’s public mental health care through NIMH funded interventions research. Washington, DC: Author.
top related