outcome measurement: put your heart into it! .hp hsp mp hsp lp hsp united way site 2 hp hsp mp hsp

Download Outcome Measurement: Put Your Heart Into It! .HP HSP MP HSP LP HSP United Way Site 2 HP HSP MP HSP

Post on 16-Mar-2019

214 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

5/12/15

1

Outcome Measurement: !Put Your Heart Into It!!

David P. Cecil, PhD, LCSW!Asbury University!

!!

Evalua&on History, Ra&onale, Literature

There is nothing so useless as doing efficiently that which should not be done at all. !

- Peter Drucker!

5/12/15

2

Evaluation: Why are we hung up?!

What are we supposed to think about evaluation? #United Way, 1996 !

What do we really think?!

Frustration!

Last minute demands! Threat of de-funding !

#(Balch and McWilliams, 1975; Wall, Busch, Koch, Alexander, Minnich, and Jackson-Walker, 2005) ! Lack of clarity! Lack of standardized process and evaluation (Mason, 2000)! Unreasonable expectations!!No matter how discouraged we get, God has not asked us to do the impossible -George Grace!

!

The Outcome Measurement Movement !Real Benefits to Real Clients!

HSPs struggling! !Need to prove worth!Need to do it quickly!!!

Benefits = Outcomes! Initial: what did the client learn?! Intermediate: how did learning change behavior?!Long-term: how did this change their condition?!

5/12/15

3

Measuring Not so bad!

Excel will do (quite nicely in fact)! Logical connections, not causal inference! Do not overreach ! Descriptive and Frequency stats!

Mean, median ! A couple good standardized instruments! Build your own!

Pre-posttests! Individualized rating scales!

Break Through!

Didnt know we didnt know (Cook and Wittmann, 1998) ! Consensus a revelation!! New language, new strategies (Mannarino and Durlak, 1980;

Bilsker and Goldner, 2002; Smith, et al, 2006; Whittaker, et al, 2006) ! Identify better AND easier strategies!

Steps in Evidence Based Practice!Chapter 2!

1. Question formulation!2. Search for evidence!3. Critical appraisal of relevant studies (must

understand statistics)!4. Determine most appropriate/effective!5. Apply intervention!6. Evaluation and Feedback!

5/12/15

4

Outcome Measurement Intended Use

Outcomes and Beyond!

Needs assessment for program development! Efficiency for bottom line! Process for process!

Formative and Program Monitoring! Outcome measurement for effectiveness

Outcomes = Benefits to clients!Addresses efficiency and process evaluation along

the way!

A User Friendly, Practical Approach!

Careful identification of resources (United Way, 1996) ! Sharpen, refine snapshot of services! Specify and Quantify (Conceptual to Operational)

Services! Those served!

Benefits to clients (Weinbach, 2005) ! Learning! Changed behavior! Improved condition!

5/12/15

5

General Evaluation Questions!

Are clients being helped?! Are clients satisfied?! Has program made real difference?! Does it deserve money spent? More, less?! New intervention better than old?! How do we improve?! Do staff make efficient use of time?!

Research on Mo&va&on in Human Services and Evalua&on

Research Ques&ons Sub-ques&ons

What are the indicators of successful implementa1on?

What factors facilitate successful implementa&on?

What are the outcomes (benefits) of implemen&ng the United Way model?

What factors are barriers to successful implementa&on?

David Cecil, PhD, LCSW (c) copyright

5/12/15

6

Organiza&onal Sample United Way Site 1

HP HSP MP HSP LP HSP

United Way Site 2

HP HSP MP HSP LP HSP

United Way Site 3

MP HSP HP HSP LP HSP

HSP: Human Service Program HP: High outcomes performance AP: Medium outcomes performance LP: Low outcomes performance

David Cecil, PhD, LCSW (c) copyright

Interview Sample United Ways

VPCI CEO TTA

Human Service Programs

PD DSP

CEO: Chief Executive Officer or President VPCI: Vice President of Community Investments TTA: Training and Technical Assistance Personnel PD: Agency or Program Director DSP: Direct Service Provider

David Cecil, PhD, LCSW (c) copyright

Indicators of successful implementation United Ways (N=9)

HSPs (N=18)

Totals (N=27)

Improved strategic planning 8 88.8% 11 61.1% 19 70.3% OM infused culture 7 77.7% 8 44.4% 15 55.5%

UW~HSP language 6 66.6% 4 22.2% 10 37%

Formalized 4 44.4% 11 61.1% 15 55.5% Can't get it out 4 44.4% 4 22.2% 8 30%

Competitive/articulate 5 55.5% 6 33.3% 11 40.7% Routine 3 33.3% 9 50% 12 44.4% Quality data 3 33.3% 7 38.9% 10 37% Community impact shift 4 44.4% 4 14.8%

Community assessment 3 33.3% 3 11.1%

Consensus 3 33.3% 3 11.1%

David Cecil, PhD, LCSW (c) copyright

5/12/15

7

Indicators of successful implementation

Low (N=6)

Medium (N=6)

High (N=6)

Improved strategic planning 1 16.6% 5 83.3% 5 83.3%

Formalized 5 83.3% 2 33.3% 4 66.6% Can't get it out 1 16.6% 3 50%

Point person 2 33.3% 1 16.6%

Forms 2 33.3% 1 16.6%

Routine 4 66.6% 1 16.6% 4 66.6%

OM infused culture 1 16.6% 3 50% 4 66.6% UW~HSP language 1 16.6% 2 33.3% 1 16.6%

Quality data 4 66.6% 3 50%

Competitive/articulate 1 16.6% 2 33.3% 3 50%

Pro-evaluation 1 16.6% 2 33.3% 1 16.6%

David Cecil, PhD, LCSW (c) copyright

Facilita&ng and Barrier Factors

High turnover

Resistant leadership

Constrained resources

Flexibility

User friendly model

Program improvement

David Cecil, PhD, LCSW (c) copyright

Outcome Measurement Review (if needed)

5/12/15

8

Outcome Measurement Limits and Process

Quasi-experimental Necessarily a convenience sample AQri&on Mostly descrip&ve stats; light use of inferen&al

Pre-post tests (baseline data compared to future data)

Mixed methods

Outcome Measurement United Way Model

Inputs- Resources Ac&vi&es- Snapshot of services Outputs- Specified and Quan&fied

Services Those served

Outcomes- Benefits to clients Ini&al, Intermediate, Long Term

Indicators- Measures outcomes Expressed as a % or # achieving outcomes

Elements of an Output

Meaningful to the client (client-centered) Services

What kind of services? How oWen and how long?

Those served Who are your clients (i.e., CMI adults) How many?

The trick: one output statement with ALL of this informa&on.

5/12/15

9

Combining Services with Those Served

Provide 1 hr/day homework assistance to 50 at-risk youth.

Provide 1hr/week home-based case management to 100 elderly adults.

Provide up to 9 months of shelter to 50 homeless men.

Inputs Activities Outputs Outcomes

Initial Intermedia

te Long-term

Staff includes contracted Occupational Therapist and Occupational Therapist Assistant.

In-home safety management

Provide in-home safety management (up to 4, 1 hour visits) to ~100 severely vision impaired clients.

Office facility/equipment

Eccentric reading classes

Provide in-home eccentric reading classes (up to 4, 1 hour visits) to ~100 severely vision impaired clients.

Adapted facility and equipment (for vision impairment)

Community re-entry classes

Provide 2 hours of community re-entry classes to ~10 severely vision impaired clients.

MUSC Partnership

Outcomes Benefits to Client

Initial outcomes Increased Knowledge Improved Skills

Intermediate outcomes Modified Behavior

Long Term outcomes Improved Condition

5/12/15

10

Inputs Activities Outputs Outcomes Initial Intermediate Long-term

Staff includes contracted Occupational Therapist and Occupational Therapist Assistant.

In-home safety management

Provide in-home safety management (up to 4, 1 hour visits) to ~100 severely vision impaired clients.

Clients increase knowledge of managing independently in their homes.

Clients utilize suggested safety plan.

Clients remain safe and independent in their homes.

Office facility/equipment

Eccentric reading classes

Provide in-home eccentric reading classes (up to 4, 1 hour visits) to ~100 severely vision impaired clients.

Clients improve reading skills using vision enhancement equipment and training.

Clients practice reading using vision enhancement equipment and recommendations.

Clients maximize potential for reading in their homes.

Adapted facility and equipment (for vision impairment)

Community re-entry classes

Provide 2 hours of community re-entry classes to ~10 severely vision impaired clients.

Clients improve community mobility skills.

Clients will negotiate community activities.

Clients will maximize ability to negotiate the community.

MUSC Partnership

Indicator(s) Target for each Indicator Data Source

Percent of clients documented as increasing knowledge on survey.

80% will demonstrate improvement on post-test.

Client safety survey

Percent of clients documented utilizing safety plan. 100% will use safety plan

Progress notes

Percent of clients and homes that meet all requirements on safety checklist.

100% meet safety checklist requirements

Safety