National Alliance to End Homelessness Family Conference
Assessment Tools Roundtable
Charlene Moran FlahertyState and Local Policy
February 7, 2008
2
CSH helps communities create permanent housing with services to prevent and end
homelessness.
CSH’s Mission
3
Family Connections
Prevent children from entering child welfare system
Reduce TANF caseload Non-recurrence of abuse and neglect Increase in stability/self-sufficiency
4
Evaluation Goals
Conscious shift from focus on outputs to investing in outcomes
Reduce the gap between knowledge and practiceWhich programs are effective?What interventions produce results? Where do we target investment?
5
Evaluation Goals
Support activities that make a real difference in lives of families
Evaluate the effectiveness of Family Connections Implement method of understanding what works
& what doesn’t Identify best practices
6
Arizona Self Sufficiency Matrix
Engagement tool Basic Assessment Evaluation
7
Arizona Self Sufficiency Matrix
Domain Score 1 2 3 4 5
Income no income Inadequate income and/or spontaneous or inappropriate spending
Can meet basic needs with subsidy; appropriate spending
Can meet basic needs and manage debt without assistance
Income is sufficient, well managed; has discretionary income and benefits
Employment No job Temporary, part-time or seasonal; inadequate pay, no benefits.
Employed full time; inadequate pay; few or not benefits
Employee full time with adequate subsidized housing.
Maintains permanent employment with adequate income and benefits.
Housing Homeless or threatened with eviction
In transitional, temporary or substandard housing; and/or current rent/mortgage payment is unaffordable (over 30%of income)
In stable housing that is safe but only marginally adequate.
Household is in safe, adequate, subsidized housing.
Household is safe, adequate, unsubsidized housing.
Food No food or means to prepare it. Relies to a significant degree on other sources of free or low-cost food.
Household is on food stamps Can meet basic food needs, but requires occasional assistance.
Can meet basic food needs without assistance.
Can choose to purchase any food household desires.
Childcare Needs childcare, but none is available/accessible and/or child is not eligible.
Child case is unreliable or unaffordable, inadequate supervision is a problem for child care that is available
Affordable subsidized childcare is available, but limited.
Reliable, affordable childcare is available, no need for subsidies
Able to select quality childcare of choice
Safety Home or residence is not safe; immediate level of lethality is extremely high; possible CPS involvement
Safety is threatened/temporary protection is available; level of lethality is high
Safety is threatened/temporary protection is available; level of lethality is high
Safety is threatened/temporary protection is available; level of lethality is high
Safety is threatened/temporary protection is available; level of lethality is high
Parenting Skills There are safety concerns regarding parenting skills
Parenting skills are minimal. Parenting skills are apparent but not adequate.
Parenting skills are adequate. Parenting skills are well developed.
8
Arizona Self Sufficiency Matrix
Children's Education One or more eligible children not enrolled in school.
One or more eligible children enrolled in school, but not attending classes.
Enrolled in school, but one or more children only occasionally attending classes.
Enrolled in school and attending classes most of the time.
All eligible children enrolled and attending on a regular basis
Adult Education Literacy problems and/or no high school diploma/GED are serious barriers to employment.
Enrolled In literacy and/or GED program and/or has sufficient command of English to where language is not a barrier to employment.
Has high school diploma/GED Needs additional education/training to improve employment situation and/or resolve literacy problems to where they are able to function effectively in society.
Has completed educational/training needed to become employable. No literacy problems
Legal Current outstanding tickets or warrants.
Current charges/trial pending, noncompliance with probation/parole.
Fully compliant wit probation/parole terms.
Has successfully completed probation/parole within past 12 months, no new charges filed.
No active criminal justice involvement in more than 12 months and/or no felony criminal history
Health Care No medical coverage with immediate need.
No medical coverage and great difficulty accessing medical care when needed. Some household members may be in poor health.
Some members (eg Children on AHCCCS
All members can get medical care when needed, but may strain budget.
All members are covered by affordable, adequate health insurance.
Life Skills Unable to meet basic needs such as hygiene, food, activities of daily living.
Can meet a few but not all needs of daily living without assistance.
Can meet most but not all daily living needs without assistance.
Able to meet all basic needs of daily living without assistance
Able to provide beyond basic needs of daily living for self and family.
Mental Health Danger to self or others; recurring suicidal ideation; experiencing severe difficulty in day-to-day life due to psychological problems.
Recurrent mental health symptoms that may affect behavior, but not a danger o self/others; persistent problems with functioning due to mental health symptoms.
Mild symptoms may be present but are transient; only moderate difficulty in functioning due to mental health problems.
Minimal symptoms that are expectable response to life stressors; only slight impairment in functioning
Symptoms are absent or rare; good or superior functioning in wide range of activities; no more than every day problems or concerns.
9
Using the Matrix
Periodicity of Data CollectionAt program entry and exitThree month interim
All adults over the age of 18 Complete by client together with case manager Score to the deficit
10
Models – Measurement
Is this assessment technique valid and reliable? Using factor analysis, the instrument was found
to measure overall self-sufficiency and 2 subscales:
Economic self-sufficiencySocial/emotional self-sufficiencyTotal composite score
11
Models – Predictive
Equations are generated from data submitted to determine the predictors of change in social emotional and economic self-sufficiency and total composite self-sufficiency score
Each individual’s predicted change is uniquely determined based upon the client’s individual characteristics
These predicted changes constitute the expected change
12
Client Data
568 unduplicated adult clients 2/3 completed entry matrix 84 clients with entry and exit matrices. 24 have been in the program 3 months 60 terminated with less than 3 months The differences between those who “stick” and
those who terminate are dramatic with longer term clients benefiting and the shorter term having minimal gains.
13
Change Entry to Exit by Time in Program
econ change
soc-emot change
total change
<3 months
3 or more
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Change Entry to Exit by Time in Program
<3 months
3 or more
14
Level of Functioning
More than 1/3 of clients either homeless or being evicted. More than ½ recently or currently homeless
Two out of every five clients had parenting skills which ranged from “inadequate to dangerous”
Approaching 1/3 of clients assessed to be functioning at a crisis level
School age children of one in seven clients not enrolled in school
15
Entry Problem Ranked
Problem Area Score Employment 1.69 Income 2.19 Food 2.21 Shelter 2.54 Childcare 2.59 Community Involvement 2.66 Family Relations 2.67 Mobility 2.90 Adult Education 2.96 Life Skills 3.34 Health Care 3.38 Parenting Skills 3.49 Mental Health 3.76 Safety 3.96 Children’s Education 4.22
16
Change in individual Clients
Change in individual clients over time initially encouraging.
3 month matrices – Typical client improved in social-emotional self
sufficiency more than a quarter (.28) of a standard deviation
– Improvement in economic self-sufficiency was even more dramatic approaching half (.42) of a standard deviation
The overall change in self-sufficiency also approached half of a standard deviation (.44)
17
Change in Individual Clients
Changes in client scores not the result of focusing on clients with fewer challenges. Rather, the lowest functioning clients made the greatest changes.
Hence, not only were the lowest functioning clients changing the most, but the increased correlations over time indicate that the impact is greater at 6 months than at 3
18
Change in Individual Clients
Entry to 6 months – The service most related to positive changes in economic self-sufficiency is rent assistance.
The best predictors of positive outcomes in social-emotional self-sufficiency are utility assistance and education and training.
Changes in overall self-sufficiency - Same three variables occur, with utility assistance having as much positive impact as the education/training and rent assistance combined.
19
Application
What are the client characteristics that are indicative that an individual is more likely to benefit from a program?
Do individual workers tend to be more effective with specific types of families?
What is the differential impact of the varying services offered by DES and contracting agencies to clients and what is their cost effectiveness?
Can we construct an “early warning system” which alerts us that based on client outcome that a worker, service delivery site or supervisor needs assistance in being effective?
20
Application
Is there an optimum time frame to work with families to maximize demonstrable effectiveness of intervention?
Is there a substantive difference between professional and paraprofessional staff in client outcomes?
How important is the time lag between first referral to FC and first interview with program staff in terms of eventual program impact?
21
Benefits/Challenges
Benefits– Client centered practice– Continuous program improvement– Target resources more effectively – Increased investment in – Incubator for new activity
Challenges– Provider buy-in– Data collection/scrubbing– Evaluation expertise– Cost
