1 external influences shaping the evaluation of a tertiary child protection program andrew anderson,...
TRANSCRIPT
1
External influences shaping the evaluation of a tertiary child
protection program
Andrew Anderson, The Benevolent Society
2
The presentation
• Background to The Benevolent Society (TBS) and its child protection program
• Background to the evaluation
• Lessons learned
• Implications for future evaluation practice
3
The Benevolent Society
• Provides support to more than 40,000 children and adults each year
• Child and family services range from childcare to child protection programs
• More than 650 staff and 800 volunteers
• 145 programs across 60 locations in NSW and Queensland
• Revenue in 2011 was $78m
• Evaluation is a strategic priority at The Benevolent Society.
4
Background to child protection program
• Three locations in Sydney since early ’90s
• Child under 12 where abuse or neglect has been confirmed and child at high risk of removal
• Detailed assessment of families and a range of interventions for children and parents
• Work with families to ensure safety of child and to address issues that led to abuse and neglect.
5
Background to the evaluation
• TBS’ first formal internal evaluation
• Identification of outcomes, development of evaluation framework, development of service model
• Outcomes included:– Improved child development and child behaviour– Improved parent-child relationship– Improved parenting– Increased safety for children within their families– Increased supportive connections with family/friends and community
• Suite of outcome measures developed – collected by staff using six monthly survey of clients from January 2007 – June 2010
• Qualitative interviews with staff and analysis of administrative data
6
Lesson 1
Choose an appropriate methodology
7
Do your homework about the program
0
10
20
30
40
50
60
70
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58
Length - Months
No
. fa
mil
ies
Terminated Families
8
What we learned
• Importance of looking at administrative data and questioning assumptions about the program– decreasing number of referrals to the program
– short term nature of the program for many families
• Outcome data is important but doesn’t mean a lot on its own
• Don’t be afraid to change approach if it isn’t working
• A mixed method approach can help avoid these pitfalls
9
Lesson 2
Choose the right outcomes
10
What we learned
• Outcomes need to be achievable and linked to clients’ length of time on the program
• Short term and medium term outcomes are important to measure as well as long term outcomes
• Importance of undertaking a program logic that doesn’t only focus on outcomes but also program theory
11
Lesson 3
Take account of the internal and external influences on the program
12
What we learned
• Interviews with program managers and staff revealed a number of external and internal influences that had affected the program:– Difficulties with staff recruitment and staff turnover
– Changing nature of referrals to the program
– Changes to child protection system
• Importance of collecting this contextual information about the program to inform interpretation of results
13
Lesson 4
Need processes in place for organisation to act on evaluation results
14
What we learned
• Unless processes are in place to actively monitor and implement evaluation results then they are likely to have little impact
• New processes and structures have improved this at the The Benevolent Society– focus on evaluation at operations meetings– development of performance improvement planning process and
link to business plans– Senior Manager research to practice role– increased focus on using data
15
What’s happened as a result
• Review of child protection programs with Australian Centre for Child protection
• Involved refining evaluation framework and revisiting service model
Child Protection Outcomes
Child is safe Child has secure and stable relationships
16
New approach to evaluation
Phase 2Evaluation planning
and piloting
Phase 5Implementing evaluation
recommendations
Phase 1Program clarification
Phase 3Implementation
Phase 4Reporting
and dissemination
17
"The only mistake in life is the lesson not learned.“
Albert Einstein
Thank you!
