Data-Driven Implementation of Triple P

Stephanie Romney, PhD

Parent Training Institute

Department of Public Heath

San Francisco, California

Helping Families Change ConferenceFebruary 11, 2011

Collaborators

� Nathaniel Israel, PhD

� Shawana Booker, ASW

� Danijela Zlatevski, PhD

Funding Agencies

� Sai-Ling Chan-Sew

�San Francisco Department of Public Health

� Debby Jeter

�San Francisco Human Services Agency

� Laurel Kloomok

�First 5 San Francisco

Presentation OverviewPresentation Overview

�Triple P in San Francisco:

�Outcomes

�Implementation

�Lessons Learned

Triple P in San Francisco

� Sept 2009: a pilot program focusing on 60 families

� Group + Pathways (families involved with Child Protective Services)

� English and Spanish

� 5 Family resource centers (neighborhood “hubs”)

� 1 outpatient substance abuse treatment facility

� 2010

� Expanded to 3 additional sites

� Therapeutic Visitation (Standard + Pathways) for families with children in foster care

Triple P Participants

263 participants served to date

� Parents to 499 children

� 0-5: 210 children

� 6-12: 182 children

Participant Ethnicityn=263

Participant Income and Educationn=263

Annual Household Income Highest Level of Education

73% of Triple P participants were unemployed at the start of the program.

Outcomes

Parenting Style

Parenting Scale

Child Functioning

Eyberg Child Behavior Inventory

Parental Stress

Depression, Anxiety, Stress Scale

Mechanism of Expected Change

Triple PEffective Parenting Practices

Improved Child Behavior

Parental Stress

Child Protective Services

Children’s Mental Health

Family Support

Organizations

Funder Interest

Parenting Scale

Strategies used by parents when disciplining children

When my child misbehavesWhen my child misbehaves……..

I handle it I handle it without without getting getting upsetupset

I get so frustrated I get so frustrated or angry that my or angry that my child can see Ichild can see I’’m m upsetupset

Example:

The Parenting Scale was administered to parents of children ages 15 months+ who had at least some contact with their child (e.g., visitation)

Parenting Scale

Scale of 1-7

� Laxness (overly permissive): � “If my child gets upset, I back down and give in.”

� Over-reactivity (overly punitive):

� “When I’m under stress, I’m on my child’s back.”

� Verbosity (talking too much / over-explaining):

� “I give my child several reminders or warnings.”

� Total Scale

� Lower scores are better

Parenting Scale Outcomes(n=114)

Differences are statistically significant for all d omains

Eyberg Child Behavior Inventory (ECBI)

� Intensity

�How often the problem currently occurs

�Never, Seldom, Sometime, Often, Always

� Problem

�Is this a problem for you? (Yes/No)

The ECBI was administered to parents of children aged 2-16 who were living with their child

ECBI Outcomes

Differences are statistically significant for both scales

Depression, Anxiety, Stress Scale

� 21 items (short form – scores are doubled)

� Timeframe: 1 week

0 Did not apply to me at all0 Did not apply to me at all1 Applied to me to some degree or some of the time1 Applied to me to some degree or some of the time2 Applied to me a considerable degree or a good part of the tim2 Applied to me a considerable degree or a good part of the timee3 Applied to me very much or most of the time3 Applied to me very much or most of the time

Example:Example:

I found it hard to wind downI found it hard to wind down

DASS Outcomes(n=126)

Differences are statistically significant

Effect Sizes

Focus Groups

� Conducted with group participants ~1 week after group completion

� Participants paid $25 (giftcard)

� Approximately 1.5 hours

Parent Focus Group AgendaI. Introductions (10 min)

II. Changes you noticed (25 min)

� in your child’s behavior

� in your parenting

III. Impressions of Group (25-30 min)

� Triple P Materials: workbook / DVD

� Usefulness of skills/strategies learned

� Homework

� Group Facilitators

� Amount of time: Number of Sessions / Length of Sessions

� Reducing Barriers to participation (food, childcare, transportation)

� Incentives

IV. Measures (10-15 min)

Changed after Year 1 to “What Was Welcoming / Unwelcoming”

V. Suggestions to improve the class (15 min)

San Francisco’s Triple P Implementation

Initial Implementation FrameworkInitial Implementation Framework

Staff Performance Evaluation

Decision Support Data Systems

Facilitative Administrative

Supports

Systems Interventions

Recruitment And Selection

Preservice Training

Consultation & Coaching

Integrated & Integrated & CompensatoryCompensatory

Fixsen, Blase, Naoom, & Wallace, 2009

Recruitment & SelectionSteps in the Readiness Assessment

�Initial contact with potential sites (usually by phone)

�Provision of practitioner kits for review

�Site completes written readiness assessment

�Meet with participants to problem-solve / answer questions prior to training

�Phone call or meeting with administrator to set budget

Triple P Site Requirements

�Attend all training & accreditation days

�2 staff + 1 supervisor trained

�Run at least 2 groups

�Start first group within 45 days

�Monthly clinical & administrator calls

�Participate in evaluation

�Provide childcare, food, transportation

Benefits to Pilot Triple P Sites

�Free training

�Free ongoing consultation / coaching

�Free practitioner resources

�Free parent resources (workbooks, DVDs)

�Financial support for running groups (~$1,500 per group for food, childcare, transportation, incentives)

Pre-Service Training

Training(20 participants per training)

# of Initial Training Days

Pre-Accreditation Workshop

# of Accreditation Days (5 participants per workshop)

Group +PathwaysGroup +Pathways 55 YesYes 44

Standard (Individual) + Standard (Individual) + PathwaysPathways

55 YesYes 44

Selected SupportSelected Support 11 NoNo 00

Initial Training

Pre-Accreditation Workshop

Accreditation~ 1 month ~ 1 month

Accreditation Process

�Quizzes + competencies (role plays)

�For Group + Pathways: 2 quizzes, 6 competencies

�All participants passed the competencies, 6 participants had to redo the quizzes

Consultation & Coaching

�Clinical calls with Triple P trainer / practitioner (1x per month)

�Run in a peer support format

� Engaging families

� Time management

� Role playing

� Homework

� Working with parents with low literacy

�Pre-Accreditation Workshop

�Accreditation: “coaching to competency”

Staff Performance Evaluation

�Outcome measures

�Fidelity measures

�Focus group outcomes

�Outcome and focus group feedback to facilitators between first and second group

�“Special topic” clinical support calls focusing on changes made between first and second group (facilitators self-evaluate)

Decision Support Data Systems

� Summarize data from multiple sources (facilitators, parents, outcome measures) in order to improve practice

� Data feedback prior to start of next group

� Access database – outcome reports by curriculum, agency, facilitator, language, demographic characteristics

Facilitative Administrative Supports

� ~$1,500 per group for childcare, food, transportation, incentives

� Free practitioner & parent resources

� Administrator calls 1x per month

Implementation Lessons

� Poor selection is costly - other components of the model will not adequately compensate for it

� Implementation data can challenges assumptions about what families need, want, and deserve

� Celebrate success – based on outcomes

� Find practitioners doing good work and highlight them!

� Bad outcomes will happen – even with an evidence-based intervention. Anticipate this and have a plan in place to turn them around

Poor Selection is Costly

The High Cost of Poor Selection

Costs2 Sites Without

Readiness Assessment (6 trainees)

4 Sites with Readiness Assessment(13 trainees)

5-day initial training($2,075 per person)

$12,450 $26,975

Training venues($208 per person)

$1,248 $2,704

Pre-accreditation workshop($152 per person)

$912 $1,976

Accreditation make-up day($1,011 per person)

Site 1: $1,011

Site 2: 0$2,022

Average cost per graduate after first Triple P group

$15,621 / 2 graduates =$7,811

$33, 677 / 32 graduates =

$1,052

Cost Per Graduate After First Group

No Readiness Assessment

With Readiness Assessment

Cost Per Graduate 1 Year Later

No Readiness Assessment

With Readiness Assessment

Graduation Rates of First Groups

No Readiness Assessment

With Readiness Assessment

Time to Start of First Group

2 Pilot Sites Without Readiness Assessments Other 4 Pilot Sites

Site 1: 95 daysSite 2: 45 days

Average: 34 days (13 - 47 days)

Challenging Assumptions:About What Families Need, Want, and Deserve

Assumption #1

African American parents need culture-specific parenting interventions

Triple P vs. “evidence-informed” culture-specific parenting

for African American parents

Triple P (n=35)

Assumption #2

Quantity of service (# of parents served) =

Quality of service (effectiveness)

The “most experienced” practitioner

What Parents Want: Focus Group Outcomes

26 Triple P Groups run to date: 26 focus groups

�Food, childcare, and transportation are essential supports, not “bells and whistles”

�Triple P materials (DVD, workbook) acceptable to parents and found helpful in both English and Spanish

�Most helpful new skills: praise, Pathways modules (anger management, misattribution) even for parents not involved with Child Protective Services

What Parents Want: Focus Group Outcomes (cont.)

� Parents requested a slower pace / more time and

� More active practice (role play)

� Strengths and Difficulties Questionnaire not understandable to Spanish-speaking parents

The focus group as a fidelity check

Q: What did you think of the videos?

Parent Response: What videos?

Celebrate Success - Based on Effectiveness

Ways to showcase effectiveness

� Public Recognition

�Last Year: 1st Annual Service Effectiveness Award (largest effect sizes)

�This Year adding the Excellence in Family Engagement Award (lowest attrition)

�Monthly workshops for new Triple P sites hosted by continuing Triple P Sites

�Outcomes presented at provider/director meetings: always have 1-page outcome summary ready!

What to do when unexpected (bad) outcomes happen

First group variability

San Francisco’s first Triple P group effect sizes ranged from -.12 to 1.36.

1st Group Effect Sizes (Parenting Scale Total)

-0.12

0

0.63

1.36

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

A B D E F G

Agency

Coh

en's

D

NA: The first groups for Agencies F & G had only one graduate each.

Have a Strategy for Addressing Poor Outcomes

� Practitioner-developed plan for the next group

� Elements to keep

� Start with strengths: “Things that went well and you want to do again in the next group” and “Things the parents said they really liked about the group”

� Elements to change

� Incorporate the practitioner’s explanation for the poor outcomes and solutions + parent suggestions to improve the class (from the focus goup)

� Always include the supervisor and administrator in the data feedback loop

Effect Sizes After 1 Year

Parenting Scale Effect Sizes After 1st Year

0.22

0.59

0.72

0.82

0.69

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

A B D E F G

Agency

Coh

en's

D

NA: After 1 year Agency G had 2 graduates.

Implementation Model Refinements

� Implementation Team Selection & Tasks

� Specify communication feedback loops / triggers for action

� Create a strategy for addressing poor outcomes before the first groups are run

� be prepared to put it into place based on the first wave of outcome data

� Consultation and Coaching

� How to determine the right intensity of coaching, step-down criteria based on outcome data

� Rapid data feedback to sites

� Before the start of the next group

� Celebrate success by showcasing effectiveness

A Few Final Thoughts

Plan implementation before training• Build implementation into Readiness Assessment

Automate as much as possible• Readiness Assessment (MS Word)• Scoring templates (Excel)• Focus Group Summaries (Excel)• Reports (Access)

� Outcomes by group, agency, facilitator, parent/child characteristics

� Summary of attendance (# and percentage of sessions completed)

A Few Final Thoughts

Reduce practitioner barriers to using Triple P• Continuing education credits for Triple P trainings• Facilitate billing for services (e.g., EPSDT

Crosswalk)

Sample Triple P Implementation BlueprintPrior to Triple P Training

Triple P Training

Post-Training / Pre-Service

During First Triple P Series / Within 1-4 months of initial training

Within 2 Weeks of the First Series Completion

Within 1-6 months of First Series Completion

Conduct Needs Assessment

Plan Triple P pilot

Initial Training (5 days for Group or Standard + Pathways)

Post-Training /Pre 1st

Triple P Series Final Prep meeting:

Class logisticsFlyer disseminationFinalize budget for essential supports

Continue weekly 1-hour consultation calls

Post-test assessments (last day of series)

Collect follow-up data (3 or 6 month)

Select Implementation Team Members and assign responsibilities based on role

Select pilot sites using Readiness Assessment

Pre-Accreditation Workshop (6-7 weeks after training)

Begin monthly problem-solving calls with Implementation Team administrators

Monthly Skill-Building Workshops

•Working with low-literacy •Role-playing•Trouble-shooting homework

Parent feedback /focus group (1 week post-series)

Conduct Data 101 Workshop: Measuring Your Effectiveness

Develop plan for evaluation and data management

Data management system must be functioning before Triple P training

Accreditation (6-8 weeks after training

Practitioners begin participating in weekly 1-hour consultation calls (2 weeks before series starts)

Continue monthly problem-solving calls with Implementation Team administrators / Identify and address site implementation problems

Data feedback to practitioners and supervisors (2 weeks post-series). Include retention rate, pre-post outcomes, parent feedback, fidelity

Present evaluation outcomes to Implementation Team

Develop Contingency Plan for Common Implementation Pitfalls

“Engaging Parents”Workshop – hosted by a continuing site (if available)

Celebrate first series completion

Recruit practitioners / sites for workshop hosting and mentorship

Develop plan for celebrating success based on performance

Celebrate accreditation Improvement plan for next series

Celebrate success based on performance

•Participant Retention•Graduation rates•Outcomes•Most improved from 1st

series to 2nd

Stephanie Romney, PhD

The Parent Training Institute

Community Behavioral Health Services

1380 Howard Street, 5th Floor

San Francisco, CA 94103

Stephanie.romney@sfdph.org

415-255-3412

For more information please see the wikipage:

http://successfulmentalhealthsystems.wikispaces.com/Triple P