measuring success using performance measurement naeh conference on ending family homelessness...
TRANSCRIPT
Measuring Success Using Performance Measurement
NAEH Conference on Ending Family Homelessness
Michelle Abbenante
Brooke Spellman
February 8, 2008
2008 NAEH Family Conference - Outcomes Workshop 2
What is Performance Measurement?
Performance measurement is a process that systematically evaluates whether your efforts are making an impact on the clients
you are serving or the problem you are targeting.
2008 NAEH Family Conference - Outcomes Workshop 3
Multiple Levels of Performance Measurement
Local Service Provider
1. Program Level Program Funding Report, such as HUD APR
CoC
2. CoC/System LevelTen-Year Plan Report Card
or CoC Application
3. State Level State-wide Report or Performance Measurement
Tool
4. National LevelNAEH Assessment of National Progress or
Federal GRPA and PART Reviews
2008 NAEH Family Conference - Outcomes Workshop 4
Why Should Programs be Interested in Performance Measurement?
We are all in the business of helping people, which means we need to…
…understand whether current activities are working to achieve intended results.
…drive program improvement and share information on effective practices with others.
…acknowledge that high-performing programs are more likely to receive funding through competitive funding processes.
2008 NAEH Family Conference - Outcomes Workshop 5
Building Blocks of Performance Measurement
Inputs include resources dedicated to, or consumed by, the program—e.g., money, staff and staff time, volunteers and volunteer time, facilities, equipment and supplies.
Activities are what the program does with the inputs to fulfill its mission, such as providing shelter, feeding the homeless, or providing job training.
Outputs are the direct products of program activities. They usually are presented in terms of the volume of work accomplished—e.g., number of participants served and the number of service engagements.
Outcomes are benefits or changes among clients during or after participating in program activities. Outcomes may relate to change in client knowledge, attitudes, values, skills, behaviors, conditions, or other attributes.
2008 NAEH Family Conference - Outcomes Workshop 6
Performance Measurement Process
ActivitiesOutreachShelters
Case ManagementRent Subsidies &
Services
Outputs# Clients Served by Program
Service LinkagesNew PSH Units/Subsidies
Vacancy Statistics
Outcomes30% exited to PH
40% increased income
25% reduction in CH25% shorter LOS
< recidivism
How do we
document our
efforts?
What did our efforts achieve?
Inputs
$ (CoC and Other)Programs
InfrastructureStaff
Should we adjust how
we spend our resources?
Should we add or change
use of resources to expand our
impact?
2008 NAEH Family Conference - Outcomes Workshop 7
Outputs vs. Outcomes
Whereas, an outcome is: An output is: Focused on what the participant
will gain from the program. Focused on what the program
will do to achieve the outcome.
A way to measure the client-level impact with clear targets and methods for measuring change.
A way to quantify the frequency and intensity of the activity.
Attributable (a result of) to that program.
Specific to the activity described for the program.
Meaningful and attainable. Feasible and attainable.
Be mindful to distinguish between outputs and outcomes.
If outcomes show the program works… outputs are needed to understand how to replicate results
2008 NAEH Family Conference - Outcomes Workshop 8
Group Exercise:Outcome, Output, or Neither?
150 clients received prevention counseling and one-time financial assistance.
90% of persons will obtain employment by completion of program.
75% of program staff will be trained in crisis management techniques.
Met 40% (50) of Permanent Supportive Housing goal.
65% of clients with chronic medical condition will improve physical health
Examples Answers
Output
Outcome
Activity
Output
Outcome
2008 NAEH Family Conference - Outcomes Workshop 9
Achieving Your Outcomes
Short-term outcomes: What change will the client experience within a month of his/her involvement in the program? How will you measure this?
Intermediate outcomes: What change will the client experience within a year of being involved in the program? How will you measure this?
Long-term outcomes: What is the long-term (e.g., 3-year) impact of the program on clients? Has it been sustained? How will it be measured?
Achieving your outcomes can be a progression. Monitoring this progression requires data that are
collected at different intervals:
2008 NAEH Family Conference - Outcomes Workshop 10
Example: Employment Program’s Impact Over Time
93% of participants (97% of people who completed the job training class) will show improved job skills
52% of participants (55% of those who complete the job training class) will obtain full-time employment
39% of participants (75% of those who get a job) will retain their jobs for > 12 months.
~95 participants will complete job training classes.
All of those who complete the training classes (~95 clients) will be referred to jobs and receive job placement counseling.
All of those who get a job (~52 clients) will receive weekly check-up calls and job counseling, as needed.
100 people expected to participate in the program annually
Short-term
Long-term
2008 NAEH Family Conference - Outcomes Workshop 11
Framework for Converting Program Goals into Outcomes
Who is the base for measuring
results?
Step 1
What do you hope to achieve
with this population?
Step 2
Within the base, how many persons achieved it?
Step 3
Who is the base population for measuring results?
Within the base, how many persons achieved it?
Step 4
Outcome (%)
1. How do I convert program goals into measurable outcomes?
2. What do I need to calculate the outcomes?
2008 NAEH Family Conference - Outcomes Workshop 12
Converting Program Goals into Outcomes: Example
Program Goal: Supporting participants in stable housing at least 6 months
Base= Persons who have been enrolled > 6 mo or have exited
(n=40 people)
Step 1
Goal: remain housed > 6 mo
Step 2
20 people are still in stable housing (6+mo) or exited after being in housing for 6+ mo
Step 3
40
20
Step 4
50% remain in stable housingat least 6 months
2008 NAEH Family Conference - Outcomes Workshop 13
Look Out for Ambiguous Concepts
For example, what do we mean by…
Developing and measuring performance outcomes often invites ambiguous concepts into the process.
• …obtaining stable housing?
• …obtaining employment?
• …increasing income?
• …accessing services?
• …becoming more self-sufficient?
Which data elements and responses will count?
2008 NAEH Family Conference - Outcomes Workshop 14
HMIS Data Elements Are the Building Blocks of Performance Measurement
Universal Data Elements: Program-Specific Data Elements:
Name Income & Sources *
Social Security Number Non-Cash Benefits *
Date of Birth Physical Disability
Ethnicity & Race Developmental Disability
Gender HIV/AIDS
Veteran Status Mental Health
Disabling Condition Substance Abuse
Residency Prior to Entry Domestic Violence
Zip Code of Last Permanent Add Services Received
Entry Date Destination
Exit Date Reasons for Leaving
Person, Program, & HH ID
* These data elements are collected at entry and exit.
2008 NAEH Family Conference - Outcomes Workshop 15
Wherever Possible Use HMIS to Define Your Concepts
• Emergency shelter • Apartment or house that you own
Based on the Destination HMIS data element, we can define stable housing (narrowly) using the following response categories:
• Transitional housing
• Permanent housing
• Substance abuse facility or detox center
• Hospital (non-psychiatric)
• Jail, prison or juvenile detention center
• Room, apartment, or house that you rent
• Don’t Know
• Staying/living with family
• Staying/living with friends
• Hotel or motel voucher paid for without ES voucher
• Place not meant for human habitation
• Other
• Foster care home or group home
• Refused
2008 NAEH Family Conference - Outcomes Workshop 16
Group Exercise
Employment Program
The goals of the program are to help unemployed clients obtain employment and help employed clients get “better” jobs. During the past year, the program served 6 (unduplicated) persons:
Client ID
Entry Date Exit Date Employment Entry
Employment Exit
1 1/31/07 9/15/07 Unemployed Employed
2 3/15/07 6/28/07 Unemployed Unemployed
3 7/11/07 -- Unemployed --
4 7/7/07 9/18/07 Employed Same Employment
5 8/2/06 5/12/07 Employed Higher Paying Job
6 11/7/06 8/2/07 Unemployed Employed
2008 NAEH Family Conference - Outcomes Workshop 17
Define the Base Population for Each Goal
Client ID
Entry Date Exit Date Employment Entry Employment Exit
1 1/31/07 9/15/07 Unemployed Employed
2 3/15/07 6/28/07 Unemployed Unemployed
3 7/11/07 -- Unemployed --
4 7/7/07 9/18/07 Employed Same Employment
5 8/2/06 5/12/07 Employed Higher Paying Job
6 11/7/06 8/2/07 Unemployed Employed
Goal 1: Achieve employment at exit
Goal 2: Obtain “better” employment at exit
Is everyone part of the target population?E.g., Do you expect to calculate an outcome for everyone?
2008 NAEH Family Conference - Outcomes Workshop 18
Calculate the Outcome for Goal 1
All unemployed persons at entry who
exited (N= 3)
Step 1
Achieve employment
Step 2
2 persons achieved employment
Step 3
3
2
Step 4
67% achieved employment
Program Goal 1: Obtain Employment at Exit
2008 NAEH Family Conference - Outcomes Workshop 19
Calculate the Outcome for Goal 2
Persons who were employed at entry and exited (N= 2)
Step 1
Improve employment
Step 2
1 person increased earnings
Step 3
2
1
Step 4
50% gained betteremployment
Program Goal 2: Improved Employment at Exit
2008 NAEH Family Conference - Outcomes Workshop 20
Exercise 2 and the Performance Measurement Process
Inputs
Money: $250,000Staff: 4 FTEs
1 Facility
ActivitiesJob Training ClassesInterview Assistance
Job Placement Services
Outputs6 enrolled in
weekly services6 employment
assessmentsReferred to av. 4
jobs each
Outcomes67% achieved empl.50% improved empl.
2008 NAEH Family Conference - Outcomes Workshop 21
Using Outcomes to Inform Future Program Operations
Step 1
Review outcomes
with program managers
Develop action steps
and timelines
Step 2
Implement action steps
Step 3
Regular monitoring
Step 4
2008 NAEH Family Conference - Outcomes Workshop 22
Step 1: Reviewing Outcomes with Managers
Program director and managers should review outcomes collaboratively to understand what the outcomes are suggesting.
Break down the outcomes to understand the underlying forces:
What’s Going On?
• What are we doing right? What activities contributed to our ability to meet/exceed our benchmarks?
• Where do we need to improve? What activities fell short of producing the desired outcomes?
• What else might be contributing to our outcomes? How can we influence or mitigate these external forces to further our positive outcomes?
2008 NAEH Family Conference - Outcomes Workshop 23
Step 2: Developing Action Steps and Timelines
Reinforcing the Good and Adjusting the Bad
Outcomes that were achieved/exceeded: Continue to support the activities that led to our positive performance.
Outcomes that were not achieved: Allocate our inputs differently to support different types/levels of activities.
Set target dates for reviewing all outcomes—e.g., 3-month intervals.
Collaborate with other service providers to “control” the external impacts on the program.
2008 NAEH Family Conference - Outcomes Workshop 24
Step 3: Implementing the Action Steps
Getting Buy-In Through Information Sharing
• You can’t implement what you don’t understand: program directors, managers and front-line staff must understand the reasons for making changes in program operations.
• Information sharing promotes the idea that “we are all in this together.”
• Information sharing is fluid: program directors, managers and front-line staff can learn from one another; it’s not a one-way (top-down) process.
2008 NAEH Family Conference - Outcomes Workshop 25
Step 4: Regular Monitoring
It’s Easier to Adjust Program Operations Incrementally than Wholesale
• Monitor your progress by generating your performance outcomes at different periods of time—e.g., 3-month intervals.
• Adjust your approach as needed, but usually incrementally.
• Important to acknowledge that clients’ needs may shift, and thus program goals and approach may also need to shift.
2008 NAEH Family Conference - Outcomes Workshop 26
Comparing Program Results
• You can compare results from one program to another to see which programs are working best and which are working least well
– With limited dollars, you want to fund the programs that are most effective.
– You can use program results to identify best practice programs and those that need TA
– You can use results from multiple programs to help set a community expectation or standard of performance
2008 NAEH Family Conference - Outcomes Workshop 27
Case Study: How Washington, D.C. Uses Program Results
• Outputs/Efficiency Measures
– Clients Served
– Chronically Homeless Served
– Occupancy: the rate at which program was used
• Interim Outcome Measures
– Permanent Housing
• Positive client destinations at exit (TH programs)
• Retain clients for 6+ months (PSH programs)
– Income: the amount of income or sources obtained
– Self-Sufficiency: change in substance use, education, mental illness or employment
• Measures apply differently to each program type and are supplemented with qualitative data for ranking purposes
2008 NAEH Family Conference - Outcomes Workshop 28
Provider Performance Measures Clients
Served Chronically Homeless
Occupancy Rate
Housing Destinations
Income Length of Stay
Self Sufficiency*
Outreach Programs
Severe Weather & Low Barrier Shelters
Temporary Shelter
Transitional Programs
Permanent Supportive Housing
Supportive Service Programs
*Self Sufficiency Temporary, Transitional and Permanent Supportive Housing Programs required to submit a Self Sufficiency Indicator will have to choose from Substance Abuse, Education, Mental Illness Assistance or Employment.
Washington, DCFY 07 DHS Performance Measures
2008 NAEH Family Conference - Outcomes Workshop 29
Apples to Apples: Risk Adjustment
• Problem: Comparing program results can encourage programs to “cream” to ensure strong results
• Solution: Risk adjustment allows you to account for differences in client populations when comparing results across programs
Results can be adjusted on the basis of…
• Client characteristics, such as demographics, family size, disability
• Client history, such as past eviction, criminal background,
• Client functionality or level of engagement/commitment to change
2008 NAEH Family Conference - Outcomes Workshop 30
Risk Adjustment Requires Expertise
• To adjust for client differences, programs need to collect consistent data on clients to use during analysis of program results
– Agree on these standards beforehand
• Develop an analysis plan for how you intend to adjust for client differences
– Engage a researcher to help develop the plan
• Even if you don’t formally adjust results, acknowledge that different programs may have different outcome expectations based on differences in clients targeted and/or served
2008 NAEH Family Conference - Outcomes Workshop 31
Simplified Illustration of Dissecting Client Outcomes on Increased Earned Income
By establishing targets, programs can be compared against CoC expectations in the future to determine if program performance is higher or lower than expected.
Program A
61% (61 clients)
All Clients (n=100)
10% (1 of 10 persons)
Disabled Clients
67% (60 of 90 persons)
Non-disabled Clients
23% (70 clients)
All Clients (n=300)
12% (30 of 250 persons)
Disabled Clients
80% (40 of 50 persons)
Non-disabled Clients
Program B
12%(31 of 260 persons)
Disabled Clients
71%(100 of 140 persons)
Non-disabled Clients
CoC OutcomeResults By Population
2008 NAEH Family Conference - Outcomes Workshop 32
System Performance Measurement
• Are your actions achieving your intended goals at the system level?
– Does the system work?
– If yes, what makes it work?
– If no, what part doesn’t work, and how do you fix it to make it work?
• Note that you may have system goals that only relate to certain types of clients or parts of the system (e.g., different goals for severely disabled persons)
2008 NAEH Family Conference - Outcomes Workshop 33
Sample Impact Measures
• Incidence of homelessness - Is homelessness declining?
• Incidence of street or CH - Is street or chronic homelessness declining?
• Length of stay in system, across all homeless programs - Do people stay homeless for shorter periods of time?
• Prevention – Are fewer people experiencing homelessness for the first-time?
• Rates of Recidivism – Are repeat occurrences of homelessness avoided or declining?
Cross-tabulate results by core characteristics to understand if/how results vary for different subpopulations
2008 NAEH Family Conference - Outcomes Workshop 34
Steps to Calculate System Length of Stay
De-duplicate clients across programs
Step 1
Consolidate sequential stays into single
episode (gaps < 30 days = same episode)
Step 4
Create table with all Entry/Exit Dates by
Client
Step 2
Calculate LOS for each Stay
Step 3
Calculate mean (168 days), low (81 days),
high (309 days)
Step 5
Client ID
Prog ID
1 A
1 B
2 A
3 C
Entry Exit Date Date
5/8/06 5/30/07
6/1/07 9/01/07
3/1/07 5/21/07
2/1/06 12/7/06
LOS
22
92
81
309
114
2008 NAEH Family Conference - Outcomes Workshop 35
Some notes of caution…
• There is more to performance measurement than conducting the analysis
– Educate, train, obtain buy-in
• Be careful about how you interpret and use the data1. Jump in, but don’t be careless in how you use the results
2. Look at the results within the context of all the outputs, interim measures and impact measures to validate the interpretation that’s being made
3. Vet the results before publicly releasing anything
4. Appropriately caveat the limitations of the data and analysis
2008 NAEH Family Conference - Outcomes Workshop 36
Summary of System Performance Measurement
ActivitiesAdjust type and
intensity of activities based on outcomes; track if
improves
OutputsDocument the level of
effort providedUse to ensure activities
delivered efficiently
OutcomesInterim Outcomes
signal client success;Impact outcomes track progress to
goals
Inputs
Use program-level &system-wide results
to adjust use of resources
Questions?
Contact us for more information or assistance:
Michelle Abbenante, [email protected] Spellman, [email protected]