11

Triple P Outcomes in CaliforniaTriple P Outcomes in California

Arizona Child Trauma SummitArizona Child Trauma SummitApril 9, 2013April 9, 2013

Cricket Mitchell, PhDCricket Mitchell, PhDSenior Associate, CiMHSenior Associate, CiMH

Summary of Breakout Session

• Overview of California’s Triple P Outcome Evaluation Data– Counties supported by CiMH

• Options to consider in developing outcome evaluation protocols

• Outcome evaluation for Arizona’s Triple P implementation – facilitated discussion

2

What is CiMH? And How is It Related to Triple P?• The California Institute for Mental Health (CiMH) is

a statewide non-profit that provides training, technical assistance, research, evaluation, and policy support to publicly-funded agencies– Supports the dissemination and implementation of 12

evidence-based practices• Program performance and outcome evaluation is a critical

implementation support

• Triple P was selected for dissemination by CiMH and promoted to county agencies in 2006– Some agencies contract with CiMH, and some do not

3

Triple P Implementation Sites Across California

CountiesMendocinoAlamedaShastaNevadaSonomaMarinSan FranciscoContra CostaSanta CruzSanta ClaraSan JoaquinVenturaLos Angeles RiversideOrangeSan Diego______________Also Tri-cities Area

4

Overview of California’s Triple P Outcome Evaluation Data

• Summer 2012 Triple P Data Submission to CiMH– Four Counties

• Los Angeles• Shasta• Sonoma• Ventura

– 74 implementation sites– 5,292 unique child clients served

5

Overview of California’s Triple P Outcome Evaluation Data

• Outcome evaluation protocols within each county vary– Data elements collected

• Demographics• Service delivery information

– Outcome measures used– Applications/software used for data entry

6

Overview of California’s Triple P Outcome Evaluation Data

• CiMH’s Program Performance and Outcome Evaluation Reports– Three primary domains

• Characteristics of clients served• Description of services provided• Outcomes achieved

– Two-Pronged Approach to Outcome Measurement» Target-specific symptoms» General mental health functioning

7

Overview of California’s Triple P Outcome Evaluation Data• Today’s presentation will highlight select

data elements from the Summer 2012 data submission– Triple P Levels and Types– Child Client Demographics

• Age, Gender, Ethnicity, Primary Language Spoken in the Home, and Primary Axis I DSM-IV diagnosis

– Triple P Outcomes• Eyberg Child Behavior Inventory (ECBI), Parenting

Scale, and Youth Outcome Questionnaire (YOQ)

8

Overview of California’s Triple P Outcome Evaluation Data – Level and Type of Triple P

9

Overview of California’s Triple P Outcome Evaluation Data – Age

10

• Range: .01 – 26.05 years– Some counties serve Transition Age Youth (15-26)

• Mean: 7.7– Standard Deviation: 4.1

• Mode: 4.0• Frequency distribution is positively skewed

– 25th percentile: 4.6– 50th percentile: 7.2– 75th percentile: 10.7

Overview of California’s Triple P Outcome Evaluation Data – Gender

11

Overview of California’s Triple P Outcome Evaluation Data – Ethnicity

12

Overview of California’s Triple P Outcome Evaluation Data – Primary Language

13

Overview of California’s Triple P Outcome Evaluation Data – Primary Axis I DSM-IV Dx*

14

*Two of the four Counties track mental health dx

CiMH Outcome Indicators• Percent Improvement

– Percent improvement from average pre-score to average post-score

• Paired t-test conducted to examine whether or not the difference is likely to be due to chance (p<.01); if not, the percent change is asterisked (*) to indicate a statistically significant improvement

• Effect Size Estimate: Cohen’s d– A standardized measure that estimates the

magnitude, or strength, of the observed change• Conventional interpretation: .8 ≈ “large”

effect; .5 ≈ “moderate” effect; and, .2-.3 ≈ “small” effect

15

CiMH Outcome Indicators• Reliable Change

– The amount of change observed in an outcome measure that can be considered an actual change, and not likely to be due to the passage of time or measurement error (p<.05)

• Complex statistical formula that takes the measure’s reliability into consideration, as well as the variability observed among scores

– Once the formula is applied, clients can be grouped into one of three categories: reliable positive change; reliable negative change; and, no reliable change

16

Overview of California’s Triple P Outcome Evaluation Data• Target-Specific Outcome Measure Focused

on Child Disruptive Behaviors– Eyberg Child Behavior Inventory (ECBI)

• Parent/Caregiver Report of the Intensity and Problematic extent of child behavior problems

• 36 items• Intensity Score Range 36 – 252

– Clinical cutpoint 131 and higher

• Problem Score Range 0 – 36 – Clinical cutpoint 15 and higher

– Used by two of the four counties

17

Overview of California’s Triple P Outcome Evaluation Data – Outcomes: ECBI

18

Eyberg Child Behavior Inventory (ECBI)

Percent Improvement from the Average Pre-

Score to the Average Post-

Score

Effect Size Estimate (Cohen’s

d)

Percent of Clients Showing Reliable Change from Pre- to

Post-

Positive Change

No

ChangeNegative Change

Intensity Raw Score

28.9%*(n=726)

[pre=135.3]1.05

63.4%

(n=460)

31.5%

(n=229)

5.1%

(n=37)

Problem Raw Score

51.0%*(n=744)

[pre=17.7]1.14

65.5%

(n=487)

29.5%

(n=220)

5.0%

(n=37)

*A statistically significant improvement, p < .01

Overview of California’s Triple P Outcome Evaluation Data – Outcomes: ECBI Intensity

19

Solid line indicates clinical cutpoint

Overview of California’s Triple P Outcome Evaluation Data – Outcomes: ECBI Problem

20

Solid line indicates clinical cutpoint

Overview of California’s Triple P Outcome Evaluation Data• Target-Specific Outcome Measure Focused on

Parenting– Parenting Scale

• Parent/Caregiver Report that assesses parenting and disciplinary styles that are found to be related to the development and/or maintenance of child disruptive behavior problems

• 30 items• Total Score is a mean item response ranging from 1 – 7

– Clinical cutpoint 2.8 and higher

– Used by two of the four counties

21

Overview of California’s Triple P Outcome Evaluation Data – Outcomes: Parenting Scale

22

Parenting Scale

Percent Improvement from the Average Pre-

Score to the Average Post-

Score

Effect Size Estimate (Cohen’s

d)

Percent of Clients Showing Reliable Change from Pre- to

Post-

Positive Change

No

ChangeNegative Change

Total Score28.0%*(n=154)[pre=3.6]

1.2548.7%

(n=75)

49.4%

(n=76)

1.9%

(n=3)

*A statistically significant improvement, p < .01

Overview of California’s Triple P Outcome Evaluation Data – Outcomes: Parenting Scale

23

Solid line indicates clinical cutpoint

Overview of California’s Triple P Outcome Evaluation Data

• General Outcome Measure of Mental Health Functioning– Youth Outcome Questionnaire

• Parent/Caregiver Report that assesses multiple dimensions of child/youth mental health functioning

• 64 items• Total Score Range -16 – 240

– Clinical cutpoint 47 and higher

– Used by one of the four counties

24

Overview of California’s Triple P Outcome Evaluation Data – Outcomes: YOQ Total

25

Youth Outcome Questionnaire (YOQ)

Percent Improvement from the Average Pre-

Score to the Average Post-

Score

Effect Size Estimate (Cohen’s

d)

Percent of Clients Showing Reliable Change from Pre- to

Post-

Positive Change

No

ChangeNegative Change

Total Score36.3%*(n=638)

[pre=63.9].74

57.5%

(n=367)

34.0%

(n=217)

8.5%

(n=54)

*A statistically significant improvement, p < .01

Overview of California’s Triple P Outcome Evaluation Data – Outcomes: YOQ Total

26

Solid line indicates clinical cutpoint

Overview of California’s Triple P Outcome Evaluation Data – Outcomes: Reliable Change

27

28

Overview of California’s Triple P Outcome Evaluation Data• Follow-up analyses of aggregate data

indicate no differences in:– change in ECBI Intensity Score outcomes;– change in ECBI Problem Score outcomes;– change in Parenting Scale outcomes; or,– change in YOQ Total Score outcomes

by gender or ethnicity

Options to Consider in Developing Outcome Evaluation Protocols• Data elements to track/collect

– Parsimony– Utility

• Outcome measures– Relevance to treatment target/goals– Psychometric characteristics (valid, reliable)– Cost– Time (administration, scoring, data entry)– Training and technical assistance

29

Thoughtful and thorough planning is the key!

Options to Consider in Developing Outcome Evaluation Protocols• Application/software used for data entry

– System already in place that can be modified? (e.g., EHR, county- or state-level information system)

– Cost– Skill level to use/employ– Utility of data elements for analysis and

reporting– Training and technical assistance

30

Options to Consider in Developing Outcome Evaluation Protocols• Frequency of analysis and reporting

– Multiple stakeholders• Different reports for different audiences

– Processes for maximizing utility of data• Clinical utility• Program improvement• Systems-level decisions

31

Feedback is Essential

Additional Considerations for Telling the Whole Story• Collect minimal data on all clients referred

– Determine entry rate– Determine additional need (waiting lists)

• Collect completion status (yes/no)– Determine dropout rate– May provide the opportunity to examine dose-response

relationships• Track clients who are served by more than one

Level/Type of Triple P (within and across providers)• Track population-level indicators (substantiated child

maltreatment cases, out of home placements, emergency room visits for unexplained child injuries)

32

Evaluation for Arizona’s Triple P Implementation – Discussion Questions• Who should be included in decision-making?• Is there an overarching evaluation framework?• What data are currently being collected?

– What additional data elements are of interest?

• What outcome measures will be used?– How will they be obtained, distributed, and used?– Who will provide training and technical assistance?

• How will data be tracked/collected from individual Triple P providers?

33

Evaluation for Arizona’s Triple P Implementation – Discussion Questions

• How will population-level indicators be tracked? With what frequency?

• Responsibility for and frequency of data analysis and reporting?

• How will data be used to inform decisions?– Client-level– Program-level– System-level

34

35

Discussion SummaryDiscussion Summary

36

The EndThe End

Contact Information•Cricket Mitchell, PhD

•Email: cmitchell@cimh.org•Cell phone: 858-220-6355