Performance Outcomes in EAP

National Behavioral ConsortiumSeptember 6, 2007Loews Coronado, San Diego

Russ Hagen, CEOrhagen@chestnut.orgDave Sharar, Managing Director dsharar@chestnut.org

Agenda

Overview of Issues with Performance Outcomes in EAP (slides 3-13)

Some example Chestnut Outcome Projects (15-28)

Where do we go as a field? (29-30) 

1.2.

3.

The Conceptual Dilemma – What is EAP?

• A “theory" of workplace intervention • A "specialized field" or profession • A "place" where other professions ply their

trade • A set of "common components" or a "practice

model" • A type of  "funding mechanism" for counseling

or mental health services• “EAPs have enlarged their scope: Can we align

metrics with relevant attributes of EAP definition?”

The Research & Empirical Base of EAP • Citations of effectiveness are often

obsolete/flawed (about 39 studies in past 10-12 years)

• Existing studies are often unpublished or proprietary

• No agreed upon markers of success, or true comparisons across vendors, models, programs

• No identified academic discipline or plan to build researchers

• Little transparency or collaboration among vendors

 

Performance is in the Eye of the Beholder

Purchaser: Are my EAP dollars being spent wisely?

Provider: (primary vendor or affiliate): Does good service or results produce rewards?

Client (employee or family member): Is the EAP responsive to my personal needs?

State of performance measurement in EAP

• Substandard performance is largely invisible

• Many measures are blunt, incomplete, distorting, exaggerated (e.g. Utilization rates)

• We lack common definitions & standard markers of success

• Under capitation, “marginal” performance receives the same rate as “optimal”

Why measure performance?

• Describe the effects or results of our interventions (outcome measures)

• Improve an aspect of the process of care (process measure), which in theory leads to better outcomes

• Make comparisons across vendors or program models

• Counterforce to deflation of EAP rates

Issues with Employers

• Many employers don’t know a good measure from a bad one

• Lack of senior management engagement• Over-reliance on consultants and brokers

who do not understand EAP• Competing goals between finance, HR,

benefits, occupational medicine, etc.• Good performance measurement is

difficult-and this difficulty is not always appreciated

Why are outcomes so difficult?

Natural Problems• Confounding factors beyond your control• Sample size too small to produce effect• Long delays when measuring over time• Low frequency of interesting outcomes

Why are outcomes so difficult?

Human Problems• Inadequate information systems• No extra funding (vendor bears cost)• Accessing employer data• Point of measurement complexity• Insufficient level of clinical detail• How does one address poor outcomes?

Criteria for selecting measures:

• Does the measure serve to enhance the productivity or well-being of employees? (e.g. is it “mission critical”?

• Is the measure based on science or opinion? (and if opinion, is there consensus)?

• Is it feasible? (resource availability, automated data collection, and statistically meaningful comparisons)

Practical Suggestions

• Implement new P4P pricing • Align better outcomes with higher

payments• Educate employers to buy based on

results, not price

Why P4P in EAP?

• Financial incentives in EAP (under capitation) are perverse, flat, almost non-existent

• Many contracts use “penalties” (withhold rather than bonus)

• So, delivering high quality EAP does not usually pay

"The larger issue is not whether EAPs are effective, but which EAPs are."

Ken Collins. "EAP Cost/Benefit Analysis: The Last Word." in EAPA Exchange (2000, Nov/Dec, p. 31). 

Started in 1985 and grew to 90 full/part time staff grossing $9 Million a year in external funds (NIH, SAMHSA, Foundations)LI-Research: Several major experiments, quasi-experiments and major surveysLI-Training and Publications: 100s of training days and largest collection of evidence-based treatment manualsEBTx Coordinating Center---Supports training, certification, and coaching of clinicians and clinical supervisors learning A-CRA and ACC GAIN Coordinating Center – supports training, certification and use of the GAIN to support diagnosis, placement, treatment planning, and research

Chestnut strategy - link researchcapability with funded EAP outcome studies

 

Types of EAP Outcomes

• General Clinical Outcomes• Work Productivity Ratings• Relationships between clinical

outcomes and workplace variables 

Global Appraisal of Individual Needs (GAIN) Short Screener (SS) embedded into WebMD's Health Risk Assessment for ADM 

• A scientifically valid, 3 minute behavioral health screener for use in general populations

• Identifies who has a disorder and who does not with 95% accuracy

• Approximates the type of problem and severity • Guides further assessment & can be used as a

measure of change

Website: http://www.chestnut.org/LI/gain/GAIN_SS/index.html

 

ADM Pilot (n=1469)

69%

77%

83%

52%

25%

21%

16%

32%

7%

17%

1%

2%

0% 20% 40% 60% 80% 100%

Internal Disorder

External Disorder

Substance UseDisorder

Total DisorderScreener

Low Moderate High

Source: Collected as part of ADM health risk assessments from 11/05 to 8/06; Total Disorder Screener is based on 14 of 20 GSS items (one item in the internal not asked, and the violence and crime screener were not asked).

ADM Pilot (n=1469) Internal and Disorder Screener Items by Total Screener Score

24%

10%

11%

8%

19%

11%

3%

2%

1%

0%

1%

16%

3%

1%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Significant problems sleeping

Feeling trapped, lonely, sad

Feeling anxious, nervous

Distressed/upset by memories

Hard time paying attention

Hard time listening

Threatened other people

You lied or conned to get things

Have you hit someone

Tried to Hide AOD Use

Fam/co-workers complain about use

Used AOD weekly

Kept using though got you in trouble

Spent lot of time getting/using AOD

Internalizing Disorder Screener

Externalizing Disorder Screener

Substance Disorder Screener

76%

90%

35%

82%

31%

15%

3%

8%

14%

17%

9%

53%

4%

7%

56%

0% 20% 40% 60% 80% 100%

Internal Disorder

External Disorder

Substance UseDisorder

Violence and Crime

Total DisorderScreener

Low Moderate High

Clinical Sample of Workers 12 Months AfterSA Treatment (n=115)

Source: ERI (Dennis & Scott) Interviews conducted between 1/05 and 5/05

$0

$3,000

$6,000

$9,000

$12,000

$15,000

$18,000

$21,000

$24,000

$27,000

$30,000

$33,000

Low (0) Moderate (1-2) High (3+)

$0

$3,000

$6,000

$9,000

$12,000

$15,000

$18,000

$21,000

$24,000

$27,000

$30,000

$33,000

Costs of Service Utilization in the NEXT 12 Months by Total Disorder Screener (n=115)

Source: ERI (Dennis & Scott) Interviews conducted between 1/05 and 5/05; p<.05

Each has a sharp right

skew

Higher Median Costs

Median Hours Absent by ADM Employees by Total Disorder Screener Score

108

156

131

0

20

40

60

80

100

120

140

160

180

200

Low (n=221) Moderate (n=123) High (n=74)

Hou

rs A

bsen

t in

a Y

ear

Based on those with any absenteeism

Next Steps with the GAIN SS  

1. Link ADM GAIN SS scores to ADM medical & pharmacy claims, workers' compensation, and other measures to determine if we can predict future claims or expenses

2. Implement a process to proactively "reach out" to ADM members with moderate or high risk scores

3. Use the GAIN SS as a longitudinal follow-up tool to measure reductions in symptoms 

ADM Formal Management Referrals in 2003

Supervisor Ratings of Work Performance (before and

after EAP) N=317 (% rated satisfactory or above)

41

82

0

10

20

30

40

50

60

70

80

90

Before EAP 3 Months after EAP

Workers' Compensation Study 2005

629,000

285,000

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

Pre-EAP1999-2001

Post-EAP2003-2005

Examined financial effect of ADM’sEAP on workers’ compensation claim dollars

Pre and Post EAP

N= 217

3 years pre- and post-intervention

55% reduction in claims

Alcohol, drugs & depression cases in 2002

Survey Data

Percent reporting dissatisfaction with . . .

12%

7%

13%

7%

14%

5%6%

3%

13%

8%

18%

9%

Emotionalstate

Socialrelationships

Maritalrelationships

Familyrelationships

J ob Kids'educationalexperience

Pre IAP After first year of IAP

22.0%

9.4%

19.0%

9.7%

31.0%

15.8%

Considered early repatdue to personal problem

Considered early repatdue to work problem

Family considered earlyrepatriation

Pre IAP

1-yearfollow up

Survey DataPercent who have considered early repatriation

Survey Data“How has your personal life been affected while living

abroad?” Percent reporting a “negative” impact.

25.00%

12.80%

Pre IAP 1 year follow up

39% decrease49% decrease

• Tension between price and performance cannot be resolved without measurement

• Obtain agreement on core set of performance measures

• All vendors/program should report on same measures

• Compete on who is best at addressing employee problems

Where do we go from here?

GOOD TOGREAT CONCEPT

BUSINESS SECTOR SOCIAL SECTORS

Defining and Measuring “Great”

Widely agreed-upon financial metrics of performance. Money is both an input (a means to success) and an output (a measure of success).

Fewer widely agreed-upon metrics of performance. Money is only an input, not an output. Performance relative to mission, not financial returns, is the primary definition to success.

Summary difference between business and social sectors thought the good-to-great framework (Jim Collins)