Quality Monitoring in the Social Services

Curtis McMillen, Ph.D.

We’ll talk about…

•  Why we need to monitor for quality in the social services, and

•  How to do it.

2

The of QualityManagement

1. Monitoring

3

What Do Agencies Monitor?

1. Monitoring … outcomes processes / fidelity adverse events / failures / errors safety consumer or employee satisfaction

community perception / reputation finances

4

Becoming A Data Driven Agency

“I have good people working here. They want to do a great job. I think they would respond well to data telling them how they are doing and how to do better. I want my agency to become a data driven agency. But I have no idea where to start, how to make that happen, what that really looks like.” -- Agency Director

5

Donabedian’s Steps to Monitoring for Quality

6

1. Figure out what to monitor 2. Set priorities 3. Selecting an approach to monitoring 4. Formulating criteria and standards 5. Obtain necessary information

6. Choose when to monitor 7. Choose how to monitor 8. Construct a monitoring system 9. Use it to bring about change

Donabedian, A. (2003). An introduction to quality assurance in health care. Oxford: Oxford University Press.

A Performance Management System A full fledged information system •  A data infrastucture –

– Thoughtful decisions on what to collect and why; –  Routinized processes to capture this data; – Ways to store and maintain and clean that data; –  People who can work the data

•  A way to turn that data into something usable. •  Processes in place that promote use of that data to

maintian high quality service and improve lower quality service. 7

The Bar Keeps Getting Raised Monitoring systems typically look for exceptions – where

things are going especially well and especially poorly.

8

Businesses now want systems that allow them to dig deeper, to explore and analyze information that sheds light on exceptions, reveals hidden trends.

Data systems behind a Performance Mgt System

9 Agency practices, client outcomes, stakeholder perceptions

Agency Database

Graphical Interface

Excel Tableau

Proprietary Expensive Big

SPSS, SAS, STATA

Data analysis platform

Electronic Client Record

Sunshine Substance Abuse Treatment

•  Operates three 90-day residential treatment homes for adults with drug addiction (serves ~ 20 each; 3 counselors, 20 shift workers)

•  Operates an aftercare program for adults from the residential program (3 caseworkers, large caseloads).

•  Operates an outpatient treatment program for adults and youth with substance abuse problems (10 counselors)

10

Sunshine at a Glance

•  37 year old private non-profit agency •  Serves about 400 clients per year •  1 Agency Director •  1 CFO •  1 Clinical Director •  16 masters level clinicians •  20 direct care staff in residential

11

Example: one metric over time

12

0

45 40

35

25

50 55

60 60 55

47

55 60

70 72 77 80 77

65

77

60

0

10

20

30

40

50

60

70

80

90

Qua

rter

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Year 2010 2010 2010 2010 2011 2011 2011 2011 2012 2012 2012 2012 2013 2013 2013 2013 2014 2014 2014 2014

% clients who would recommend Sunshine

Adding timeline information

13 Agency-wide metric

caseworker metric

0

45 40

35

25

50 55

60 60 55

47

55 60

70 72 77 80 77

65

77

60

0

10

20

30

40

50

60

70

80

90

Qua

rter

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Year 2010 2010 2010 2010 2011 2011 2011 2011 2012 2012 2012 2012 2013 2013 2013 2013 2014 2014 2014 2014

% clients who would recommend Sunshine

New clinical director

Drilling down to program

14

0

10

20

30

40

50

60

70

80

90

Qua

rter

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Year 2010 2010 2010 2010 2011 2011 2011 2011 2012 2012 2012 2012 2013 2013 2013 2013 2014 2014 2014 2014

aftercare

all

rescare

Outpt

% of clients who recommend

Agency-wide metric

caseworker metric

New clinical director

Drilling down to caseworker

15

45 40

35

10 15 15

20 20

10 10 10

25

55

65 65 65 70

50

60

25

60 60 60

20

30

45 50 50

30 30 30

50

75

60

70 70 70 70 70 70

25 20 20

5 10

0 5

10

0 0 0

25

85 85

70 75

80

50 50

0

50

40 35

5 5 0

5 0 0 0 0 0

5

50 55

50

60

30 30

5 0

10

20

30

40

50

60

70

80

90

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

2010 2010 2010 2010 2011 2011 2011 2011 2012 2012 2012 2012 2013 2013 2013 2013 2014 2014 2014 2014

Recommend to friend: Aftercare Program

All AC

Marge

Toni

Mark

Megan

Agency-wide metric

caseworker metric

New clinical director

A How question: Where does the data come from in a performance

management system? chart reviews electronic client record reports adverse event reports self-report surveys observations

16

Why do we monitor?

Cynic’s version •  to get or remain accredited or licensed •  to satisfy a funder •  to satisfy the gods of bureaucracy

17

Why do we monitor?

Idealist’s version •  we want to see whether something important is getting

better or worse •  we want to see WHO is doing better or worse •  we want to see WHERE things are better or worse

18

The e of QualityMonitoring

We monitor to become a data driven agency. We monitor to know what to emulate, what to fix, what

needs our attention, where things are going off track. We monitor for accountability

19

Keys to Success

•  Get proper sponsorship and resources for the monitoring effort.

•  Create the right metrics •  Design a compelling interface •  Plan ahead to ensure end use adoption

»  Eckerson, 2011, Performance Dashboards, 2nd ed.

20

Getting the metrics right: What to monitor

•  Stuff that is critical – if things go wrong here, important stuff is at risk.

•  Stuff that happens frequently. •  Stuff that is “hands on” – where humans make

decisions. 21

KEY QUESTIONS: How much stuff do you monitor? What do you put on your dashboard / scorecard?

22

Getting the metrics right: What to monitor

QualityTool: Balanced Scorecard

•  The top top things the agency/ program wants to monitor.

•  Often want from myriad perspectives: financial, consumer, internal process perspective, innovation, etc.

•  Balanced because an improvement in one area can have negative consequences on another.

•  Can be used for accountability at multiple levels.

23

Let’s Start with what we know

•  Pick key metrics for SSA •  Strive to capture multiple perspectives •  Strive for balance.

What are your top five?

24

What do you monitor at Sunshine Substance Abuse Services? Outpatient substance abuse counseling Residential Substance Abuse Counseling After care Substance Abuse Counseling

25

Back to Sunshine

There are lots of good choices

•  relationship/sensitivity in delivering services •  getting it right, competence, knowledge about substance

use, •  staying sober – amount of use. •  agency finances / payor mix

•  employee satisfaction •  Audit perspective – medicaid compliance

26

5-Year Performance Dashboard for Sunshine

27

0%

20%

40%

60%

80%

100%

1 3 5 7 9 11 13 15 17 19

Few

Mostly

Fully

Medicaid Compliance

0 20 40 60 80

100

Qua

rter

Q

1 Q

2 Q

3 Q

4 Q

1 Q

2 Q

3 Q

4 Q

1 Q

2 Q

3 Q

4 Q

1 Q

2 Q

3 Q

4 Q

1 Q

2 Q

3 Q

4

Year 2010 2010 2010 2010 2011 2011 2011 2011 2012 2012 2012 2012 2013 2013 2013 2013 2014 2014 2014 2014

Would recommend Sunshine

0

50

100

1 3 5 7 9 11 13 15 17 19

Payor Mix

0

20

40

60

80

1 3 5 7 9 11 13 15 17 19

AC

OP

% sober 6 mos after tx start

Satisfied employees

Medicaid grant

No pay

Ramifications of Managing by these Foci

28

0%

20%

40%

60%

80%

100%

1 3 5 7 9 11 13 15 17 19

Few

Mostly

Fully

Medicaid Compliance

0 20 40 60 80

100

Qua

rter

Q

1 Q

2 Q

3 Q

4 Q

1 Q

2 Q

3 Q

4 Q

1 Q

2 Q

3 Q

4 Q

1 Q

2 Q

3 Q

4 Q

1 Q

2 Q

3 Q

4

Year 2010 2010 2010 2010 2011 2011 2011 2011 2012 2012 2012 2012 2013 2013 2013 2013 2014 2014 2014 2014

Would recommend Sunshine

0

50

100

1 3 5 7 9 11 13 15 17 19

Payor Mix

0

20

40

60

80

1 3 5 7 9 11 13 15 17 19

AC

OP

% sober 6 mos after tx start

Satisfied employees

Medicaid grant

No pay

How often do you monitor?

29

How often do you: •  Conduct chart reviews? •  Mount Surveys? •  Run record reports?

How often do you monitor: •  Outcomes? •  Consumer satisfaction? •  Employee satisfaction •  Compliance? •  Payor mix? • Relationship/ Respect • Promptness Each service encounter? Monthy? Quarterly? Annually? :

My generic big three

•  Respect •  Competence •  Helped

30

How often do you ask these questions? At each service encounter? Monthy? Quarterly? Annually?

Ways to get to the right metrics

•  Know your agency/program’s mission, vision and values •  Destination statements •  Strategy Maps •  Strategic Themes •  SWOT analysis (Strengths, Weaknesses, Threats,

Opportunities) Unique to Social Services? •  Logic models

31