1 ichh learning and evaluation framework interagency council on housing and homelessness framework...

1 ICHH Learning and Evaluation Framework

Interagency Council on Housing and Homelessness

Framework for Learning & Evaluation

May 20, 2009


Meeting Agenda

Purpose:

Agree on an evaluation &

learning framework for

ICHH Networks.

Agenda

1. Welcome and Introduction (One Family)

2. Overview

3. Outcome Evaluation

4. Innovation Evaluation

5. Network Evaluation

6. Roles, Resources and Timelines

7. Next Steps


Steering Committee

• Bob Pulster, ICHH• Sue Beaton, Fireman Foundation• Deborah Fung, Fireman Foundation• Dennis Culhane, University of Pennsylvania• Emily Cohen, One Family• Joe Finn, MHSA• Madeleine Taylor, Innovation Network for

Communities• John Cleveland, Innovation Network for

Communities


Context Overview


The Stakes Are High

• The ICHH funding is one-time funding designed to push a paradigm shift in the state approach to homelessness.

• Innovation in the field will drive future state policy around know best practice.

• We need a rigorous evaluation and learning processes to make a convincing case for systems reform.


The Paradigm Shift

Shelter

Day CareEmployment Assistance

Family Supt Services

MH/SA Services

Prevailing Model Emerging Model

Housing Stabilization

Prevention, diversion & rapid re-housing

Day Care

Employment Assistance

Shelter

Family Supt Services

MH/SA Services

Housing Placement


Systems Transformation

• From “the Continuum” to “the Network”• Creating a New Community of Practice:

Housing Stabilization• New Service Priorities:

– Diversion– Tenancy Sustainment– Service Coordinators– Benefits Counselors– Housing Relocation Specialists– Family Mediation– Home Visitors


ICHH Evaluation Mandates in 5 Yr Plan

• Improve statewide data gathering systems.

• Create an information management system to monitor performance and outcomes.

• Assess progress on key benchmarks and use information to make system adjustments.


Evaluation Overview


Three Areas of Evaluation

Purpose: To understand what works and doesn’t work to reduce individual and family homelessness, and align state

policy with this knowledge.

Areas of Learning

Outcomes

What were the targeted outcomes and to what extent

were they achieved?

Innovations

What innovations were used, and how did they affect the

outcomes?

Networks

What is the structure of the networks, and

what effect did it have on program

outcomes?


Three Areas of LearningArea What We Want to Learn

Outcomes • What were the targeted outcomes (impact on homelessness)

• To what degree were these outcomes achieved?

Innovations • What practice innovations did the networks implement?

• How were these practices different from traditional practice?

• What impact did the practice innovations have on outcomes?

Networks • What is the structure of the network?• How is the network governed and managed?• How well is the network developing trust and

creating value for members?• What is the relationship between the network

structure and innovations?


Three Processes

Process What it Involves

Regional Network Processes

• Regions develop systems for collecting common data elements

• Regions collect outcomes data• Regions assess their innovations and

networks

Technical Assistance Support

• MHSA, One Family and Innovation Network for Communities provide TA to regional networks

• Site visits; interviews; phone consultations

Peer Learning • Network representatives meet quarterly to share best practice learning in specific areas


Measuring Outcome Data


Collecting Outcome Data

• ICHH’s goal is to implement an improved state-wide HMIS system.

• In implementing HMIS, regions can utilize existing platforms if they meet appropriate data exchange standards.

• However, outcome data planning and collection cannot wait for the completion of the HMIS system.

• All regions will be asked to collect “common data elements” to evaluate outcomes.


Common Data Elements• HUD has issued new HMIS standards for the

Homelessness Prevention and Rapid Rehousing Program (HPRP)

• DHCD will adopt these new data elements for HPRP providers and regional networks

• New elements include 1) tracking HPRP eligibility, 2) services provided, and 3) outcomes

• Client tracking and recertification for assistance until exit

• Quarterly reports due to HUD and DHCD starting January 2010


Outcome Evaluation Resources

• MHSA will work with each network on implementation of the “Home and Healthy for Good” evaluation methodology

• Dennis Culhane will serve as an overall consulting resource to ICHH on outcomes data collection and analysis

• One Family will provide TA to networks on evaluation of family outcomes


Home and Healthy for Good Methodology

• Secure informed consent• Data collection through one-on-one interviews:

– Entry Interview, Follow-Up Interviews, and Exit Interview

• Entry of data by MHSA into secure databank• Analysis of data:

– Costs

– Medicaid

– Quality of life

– Sub-analysis

• Comparison of data across networks


Assessing Innovations


Common Types of Innovations

Early Warning Systems

Prevention and Diversion

Assessment and Intake

Case Planning & Management

Access to Housing

Supportive Services


Common InnovationsType of Innovation Elements


• Training for risk factors across the network• Development of alert systems

Prevention & Diversion

• Tenancy preservation & eviction prevention services• Stabilization teams; Legal services and mediation• Direct diversion to housing, bypassing shelter system

Assessment & Intake • Mobile assessment• Pre-assessment tools; uniform assessment tool• Case triage and targeting tools• Common databases

Case Planning • Comprehensive case planning (Individual Service Plans)• Case management and case coordination

Access to Housing • Rapid re-housing; shallow subsidies• Low threshold housing• Housing location systems; housing searches; housing database

management• Landlord network and relationship management• Housing development

Supportive Services • Flexible funding for short-term and shallow subsidies• Employment and training; Work First; employment counseling• Financial management training


Things To Learn About Innovations• What specific practices did the innovation consist of?

• How did the practice vary across regions? Why?

• How did the practice vary across populations? Why?

• How did the innovation affect outcomes?

• Were networks used in the innovation? If so, what kind?

• How critical was the network design to the innovation success?

• What are the “best practices” in each category?

• What are the economics of each innovation? How much do they cost and what do they save?

• What are the policy implications?


Innovation Evaluation Process

• One Family and MHSA site visits to document actual innovation practices

• Debrief with network staff on details of implementation: What practices were actually implemented? What made a difference in outcomes?

• Development of best practice examples for peer learning sessions


Network Evaluation


Network Evaluation

Area of Focus Points of Inquiry

Network Structure and Composition

How is the network structured and how are members connected to each other?

Network Governance and Management

How does the network make decisions and manage its work?

Network Production

What does the network actually do? What services does it deliver?

Network Health What is the level of trust between network members? How highly do members value the network?


Innovations and Networks

Type How A Network Model Might Change Practice


• Common training across a broad range of network members

• Common risk factors and alert systems

Prevention • Collaboration on prevention across the network

• Joint interventions

Assessment & Intake

• Common intake information and triage processes

• Common data systems

Case Planning

• Network-wide case managers• Common ISP formats

Diversion • Sharing of diversion resources• Collaborative case management on diversion process

Access to Housing

• Network collaboration on housing searches and location systems

• Common landlord relationship management and standards

Supportive Services

• Involvement of service providers in the network• Preferred access for network clients


Different Types of Networks

CONNECTIVITYInformation

ALIGNMENTIdentity

PRODUCTIONInitiative

Connects people to allow easy flow of and access to information and relationships

Aligns people to develop and spread an identity and collective value proposition

Individuals come to share a set of ideas, language, or standards

Fosters joint action for specialized outcomes by aligned people

•Mobilization•Advocacy•Learning and dissemination•Delivering products and services

Increased levels of complexity


Different Networks for Different Purposes

Example: Early Warning System

•Connecting – Shared alert system

•Aligning – Common standards on risk factors

•Producing – Joint interventions


Network Evaluation Process• Develop network baseline map

– Material reviews

– On-line survey

– On-site interviews or focus groups

– Develop electronic map

• Develop baseline assessment– Develop network scorecard

• Collect follow up data in 9 months and at completion


Network Evaluation Products

• Graphic network maps that show baseline and evolving structure

• Network scorecards

• Case studies

• Network evaluation for each region

• Summary network evaluation


Peer Learning Sessions


Peer Learning Sessions

• Quarterly meetings (Up to 6 over the 18 month time frame)

• 1-4 people per region (up to 50 people total)• Half day; facilitated by the Innovation Network for

Communities• Agenda will include:

– Specific case studies on the focus topic

– Presentation of outside points of view

– Round-robin discussion on best practices

– Identification of challenges and barriers

– Policy options


Roles and Resources


RolesWho What

Regions • Develop evaluation plans• Collect and analyze outcome data• Participate in site visits; share information• Participate in peer learning

ICHH • Establish common data elements• Aggregated data and prepare ICHH reports

Dennis Culhane • Assist in design and analysis of common data

MHSA • Support regions in HHFG evaluation• Provide TA on individual innovations

One Family • Support regions in family outcome evaluation• Provide TA on family innovations• Convene peer learning sessions

Innovation Network for Communities

• Collect data for network evaluations• Provide TA to regions on network design• Facilitate peer learning sessions


Evaluation Plan Elements

1. Who is responsible for overall evaluation strategy.

2. Strategy for collecting and analyzing outcomes data and connecting with HMIS system.

3. Strategy for documenting and assessing specific innovation practices.

4. Strategy for network evaluation.

5. Evaluation timeline and budget.


Resources

• Evaluation and learning resources being provided by the Fireman Foundation

• $25,000 grants to each network to support data collection; participation in site visits; and participation in peer learning

• One Family support provided free of charge

• Separate TA and evaluation contracts

• Some ICHH resources to support peer learning process


Timeline Elements

What When

Regions Submit Evaluation Plans July 1, 2009

Baseline Site Visits July, 2009

Follow Up Site Visits Q4, 2009

Peer Learning Events Quarterly, beginning in July, 2009

Outcome Data Collection Monthly, beginning

Interim Report to ICHH ?

Final Report to ICHH ?


Summary of Key Points• Every region will need an evaluation plan. The plan will have three

components:– Outcome evaluation and common data elements

– Evaluation of innovations

– Network evaluation

• In addition to the development of the plan, participating in the evaluation process will involve:– Participating in interviews with evaluators

– Responding to information requests (such as on-line surveys)

– Analyzing root causes; participating in peer learning processes

• ICHH has organized resources to support this process.– $25,000 grant to each network

– Technical assistance resources


Additional Questions and Discussion

1 ichh learning and evaluation framework interagency council on housing and homelessness framework...

Documents