the importance of measuring outcomes
Post on 14-Jun-2015
271 Views
Preview:
DESCRIPTION
TRANSCRIPT
The Importance of outcomes and evaluation in service delivery
Yfoundations 10th July 2012
Prof Ilan Katz (SPRC)
Will cover
Why outcomes are important Evaluating for outcomes Monitoring and evaluation Types of evaluation Some key methods Conclusions
Why are outcomes important
Most services focus on outputs: Numbers of clients seen Numbers of service events Unit costs
However this does not measure impacts ie whether the service makes a difference can create perverse incentives.
Outcome measurement can improve service delivery.
How does evaluation improve services
Important to know what impact the service is actually making on clients and why.
Better understanding of who benefits most from the service and which aspect of the service is most beneficial.
Results are fed back into the service so that lessons can be learned about elements of effective practice.
Outcomes and accountability
Accountability increasingly focused on outcomes rather than outputs.
Funding decisions now rely on cost effectiveness as well as assessment of need.
Outcomes and evaluation can also increase accountability to service users, as participants and recipients.
Evaluation and monitoring
Monitoring – regular collection of data which is used for quality assurance, adherence to standards and compliance.
Evaluation – one off or episodic activity focusing on effectiveness of implementation, processes, outcomes and/or cost effectiveness.
Both are important and can build on each other.
Theory of change or program logic
Links inputs processes, outputs and outcomes.
Explains why activities should lead to particular outcomes.
Needs to be backed up by theory and empirical evidence and be specific.
Explains the link between inputs, processes and outcomes.
Process evaluation
Is service doing the things it is supposed to do in the most efficient way possible?
Has the program been implemented as planned?
Is it targeting the right people? What are the barriers to accessing the
service? How well is it working with other services? What are clients’ and other stakeholders’
views of the service?
Intermediate Outcome:Young people at risk of homelessness are provided with safe accommodation and given the
opportunity to address issues leading to homelessness
Ultimate Outcome:Young people find stable accommodation and are able to participate actively in society to their maximum potentioal
Situation· Limited access to
Community specialist services
·High demand for service
·Young people unable to access mainstream housing services
·Young people leaving families and becoming vulnerable to exploitation and abuse
Inputs$X put into youth homelessness · Staffing resources
Policies – Commonwealth and NSW
Other services
Client Group: Young people at risk of homelessness
Inputs$X put into youth homelessness · Staffing resources
Policies – Commonwealth and NSW
Other services
Client Group: Young people at risk of homelessness
OutputsWhat services will provide:· counselling·initial assessments·Temporary accommodation·brief interventions·Benefits advice·Peer support·providing education and support to referrers
Who the services will interact with:·Young people with acute housing issues ·families and carers·other members of the broader youth services·primary referrers and other service partners
OutputsWhat services will provide:· counselling·initial assessments·Temporary accommodation·brief interventions·Benefits advice·Peer support·providing education and support to referrers
Who the services will interact with:·Young people with acute housing issues ·families and carers·other members of the broader youth services·primary referrers and other service partners
ImpactsFor children and adolescents:· Improvement in housing situation ·Safety·Improvements in educational/vocational participation·Improvements in social functioning·Improvements in financial stability·Reduction in wait time for service·Better interaction with family ·Improvement in health and mental health
For homeless Service System:·Lower numbers of homeless youth·Better inter-agency collaboration·Longer term engagement with vulnerable clients
For Referrers:·Clear referral pathway·Improvement in collaborative care
ImpactsFor children and adolescents:· Improvement in housing situation ·Safety·Improvements in educational/vocational participation·Improvements in social functioning·Improvements in financial stability·Reduction in wait time for service·Better interaction with family ·Improvement in health and mental health
For homeless Service System:·Lower numbers of homeless youth·Better inter-agency collaboration·Longer term engagement with vulnerable clients
For Referrers:·Clear referral pathway·Improvement in collaborative care
9
Methodologies
Administrative data (employment, health, housing)
Worker assessments, case files Use of standard validated surveys
Self completed surveys pre and post questionnaires Satisfaction, wellbeing, circumstances
Qualitative methodologies/action research
Outcomes
Immediate Satisfaction, appropriate referral
Intermediate Improved wellbeing, skills, housing
Long term Home ownership/stability, employment, social
networks
Challenges of measuring outcomes
When? Should you follow up clients?
How? Questionnaire, feedback, admin data
What? How to define a good outcome
Comparison and benchmarking Attribution
How do you know it was your service making the difference?
Who should do the evaluation?
In house Advantages
• Cheaper and more tailored to agency needs.
Disadvantages• Lack of credibility and independence.• Need organisational expertise
External Advantages
• Independent, authoritative – provides feedback
Disadvantages• Expensive and potentially burdensome, potential
embarrasment
Ilan KatzSocial Policy Research Centre
Matthew GrayCAEPR, ANU
Ilan.katz@unsw.edu.au
www.sprc.unsw.edu.au www.anu.edu.au/caepr
G2 Western CampusUniversity of New South Wales Kensington 2052NSW, Australia+61 2 9385 7810
top related