influencing the use of evaluation findings how four state government organisations are working...
TRANSCRIPT
Influencing the use of evaluation findings
How Four State Government Organisations Are Working Towards Developing a Utilisation Culture
Introducing the Panel
•Anna Georgalis (Deborrah Jepsen) – Department of Business & Innovation
•Quynh-Tram Trinh – Department of Human Services
•Sue Doyle – Arts Victoria (Department Premier & Cabinet)
•Kirsty Fenton – Department of Primary Industries
Today’s Presentation will focus on:•Developing an organisational culture of evaluation through utilisation of evaluations
•Using evaluation evidence to support unpopular decision
•Factors and approaches that support the utilisation and influence of evaluation findings, such as:
• Methods that promote ownership of evaluations to increase the likelihood of implementation of findings and recommendations
• Establishment of a strategic evaluation agenda• Organisational maturity
Structure
Panel members – 30 mins
Questions & Discussion – 30 mins
Sharing approaches, challenges, ‘hot tips’ for developing a culture that supports the utilisation and influence of evaluation at all levels of an organisation
• Department of Business and Innovation
• Anna Georgalis
• Victoria’s lead economic, business and innovation agency
• Role is to continue to develop and promote policies and actions that increase productivity and participation which supports sustainable growth
Department of Business and Innovation (Victoria)
About the Victorian Department of Business and Innovation
Department of Business and Innovation (Victoria)
• Non-recurrent nature of program funding required recurring business case development
• No evidence to support why the initiative was important
• Increasing demand for accountability and transparency from:
– Ministers
– Victorian Auditor General
– Central Agencies
• New recent wave of strengthened focus from Central Agencies
The drivers for evaluation
Department of Business and Innovation (Victoria)
• Departmental Evaluation Framework and Policy that specifies:
– Roles and responsibilities
– Quality assurance
– Processes and timing
• An Initiative Design Framework for business case development that includes a focus on evaluation at the design phase
• The use of Investment Logic Maps (or equivalent) and development of evaluation or performance monitoring frameworks
• A central evaluation unit dedicated to supporting evaluation across the Department
Key approaches and tools in developing an organisational evaluation culture
Department of Business and Innovation (Victoria)How evaluation influences the development of strategies, policies and programs
Continuous Improvement Cycle
Department of Business and Innovation (Victoria)
• Increased understanding, capability and utilisation of evaluation across the Department
• Stronger business cases
• Relevant and appropriate new programs based on evaluations and stronger evidence
• Evaluation used to redesign programs or cease programs as necessary
• Culture of continuous improvement
• Increased accountability and transparency
Successes so far
Department of Business and Innovation (Victoria)
• Must have leadership direction from the top, authorising environment and critical mass of supporters
• Ensure engagement with stakeholders throughout
• Use authorising environment to your advantage
• Rigorous and ‘fit-for-purpose’ methodologies for evaluation
• Critical thinkers and analytical capability
• Sustained vigilance
• Bringing about change, especially using evaluation, is challenging
Lessons learnt
Evaluation & Influence in Government
Quynh-Tram Trinh, Manager, Research & Evaluation Unit
Department of Human Services, Victoria
12
Presentation outline
1.Centralisation of Evaluation Capacity
2.Authorisation for central R&E unit
3.Aspects of Influence
13
Centralisation of Evaluation Capacity
July 2010 – R&E Unit nominally centralised. It pulls together existing evaluation EFTs in:
• Housing & Community Building division (social housing & homelessness)
• Children, Youth & Family Services division (child protection, youth justice and family services)
• Small amount of additional resourcing in research
14
Authorisation for central R&E unitThe R&E Unit is authorised to:
• To provide strategic R&E leadership and improve the evidence base to support effective policy, program and service development in DHS; &
• To promote a collaborative and strategic approach to R&E in areas of priority for DHS.
R&E Unit sits in Policy & Client Outcomes Branch, IWS Division of DHS, Victoria. P&CO branch’s objective is:
To improve outcomes for people who many need the support of the human services system through high quality strategic policy,
research & evaluation
15
Aspects of Influence
• Day-to-day
• Strategic
16
Aspects of Influence – Day-to-day
• Active advice on high-quality evaluation approach e.g. objectives, questions, methodology, contract management, public dissemination
• Direct contract management on high-profile evaluations. Practise high-standard stakeholder management, contract management, ensure methodological rigour, good communications & public dissemination
17
Aspects of Influence – Day-to-day
• Facilitation of logic mapping sessions to produce strategic & logical frameworks
• Beginner’s workshop on program logic
18
Aspects of Influence – Strategic
• Early 2011 - R&E Audit & benchmarking - Developing guidelines
• Mid 2011 – R&E Strategic Agenda development - R&E Capacity building strategy
• Late 2011 – R&E Strategic agenda endorsed - Departmental governance
arrangement for all R&E resources
• 2012 – Implementation of strategic agenda
19
Influencing the Use of Evaluation Findings
Sue DoylePolicy and ResearchArts Victoria
• State government body which advises on and implements arts policy.
• Charged with making the arts available and accessible to all Victorians and with supporting and developing Victoria's artists and creative industries.
Arts Victoria
Rawcus Theatre Company – performers with and without disabilities
Survey of past evaluations
• How useful was the evaluation, how has it been used?• Were findings used or recommendations implemented? If
not, why?
Results Where recommendations were made, over half implemented Most useful in guiding program improvements &
communicating value of program, including lobbying for more resources
One issue with some recommendations (aside from lack of resources) is lack of understanding about operating environment of programs
l
Survey of past evaluations
• Ideas for improvement
Actionability & relevance of recommendations sometimes an issue
Could be addressed through evaluator providing findings & then facilitating discussion amongst key stakeholders (including senior executive and program staff) to develop recommendations or actions
• Greater ownership of evaluation process and findings by influencers and decision makers
• Still need political will and available resources
ExampleEvaluation of Castanet
• Castanet – community arts network• Castanet evaluated using more “action-research”
based approach• Dynamic relationship between findings and
program/initiative, increases likelihood and/or timeliness of addressing findings
• Anonymous feedback mechanism for network • Evaluator as advocate
Department of Primary Industries
Kirsty Fenton
“The systematic collection and use of information to improve decision making, demonstrate impact and performance of services to the public and key stakeholders, and enhance organisational learning”
If evaluations aren’t used, how can we expect them to be
influential?
While evaluation within FSV is implicitly utilisation-focussed, how evaluations could be used outside the immediate project is often not given the appropriate consideration when designing an evaluation.
Sometimes, evaluations are considered by staff as a compulsory project deliverable, but not as a vehicle for learning.
It is uncommon for evaluation findings and recommendations to be shared with other teams, or their findings utilised to inform policy and strategy.
Lack of formal process to share evaluation findingsLack of a central, searchable repository Project deliverable vs vehicle for learningVarying levels of quality & consistency in evaluation planning, implementation, reporting and utilisation.
Utilisation Scorecard
Evaluation
Findings
Improvement Inform Accountability
Own Project
Delivery
Other project
delivery
Investment
decisions
Strategy:
Change or confirm
Demonstrate
achievement of
outcomes and
value for $
New project
design &
development
Evaluation design &
implementation
(other projects)
X X X
Step 1: Understanding your project
1.1 Program logic
Step 2: Evaluation planning
2.1 Identify the evaluation audience
2.2 Define the purpose of the evaluation
2.3 Define the focus of the evaluation
2.4 Developing an evaluation plan
2.5 Key Evaluation Questions
2.6 Methods
2.7 Analysis
Step 3: Evaluation Implementation
3.1 Conducting the evaluation
3.2 Reflection
Step 4: Reporting
4.1 Reporting
4.2 Utilising evaluation findings and recommendations
4.2.1 Evaluation Communication Plan
4.2.2 Recommendation response and prioritisation
4.2.3 Evaluation summary
Templates
T1: Evaluation audiences
T2: Roles and responsibilities
T3: Evaluation work plan
T4: Evaluation communication plan
T5: Recommendation response and prioritisation
T6: Evaluation summary
Evaluation Summary
Captures important information, such as evaluation purpose, key findings, recommendations and reflections.
Summary sheetDetailsBranchKey Project CMI#MIS#Project team contactEvaluation team contactEvaluation titlePurpose of the evaluationCollaborators (eg. other PAG, Dept.)Completion dateContractor name (if applicable)
Summary sheet – cont.Project Outcomes-Key findings
Utilisation & recommendations-Have recommendations been developed and/or actioned? (if yes, add recc response & prioritisation template 5)
ReflectionEvaluation outcomesEvaluation process
Evaluation Outcomes•Are the findings relevant to other programs•Insights into:
• Strategy – are we doing the right thing?• Design – is our project/service design right?• Delivery – how well did we implement the project/service?
Improvements?
Evaluation Process•What went well? •What was frustrating? •What might you do differently next time? •3 things you’d suggest to someone else undertaking evaluation
Individual project level – increased utilisation by commissioner and project team, use of previous evaluations by other project teams
Key Projects – draw together findings and recommendations from multiple projects
Sector Service Evaluations
(Summit Workshop – KPM’s, branch directors, A&S, SET, project & evaluation managers – use also for FSV mini-conference, AFG evaluation & investment conferences)
Service Design branch
Strategy & Investment branch
- Analysis & Strategy – to inform policy & strategy
- Strategic Evaluation – meta-analysis & meta-evaluation (to inform ECB activities)
- KPM’s etc. for reporting & learning
Next Steps
Trialling summary sheet with project teams
Tailored training
Incorporation of evaluation into project management cycle
Influencing utilisation – take home messages
• Leadership & direction from the top, authorising environment • Support and ownership from decision makers and influencers
is crucial – participatory approaches encourage this• Ensure engagement with stakeholders throughout• Use authorising environment to your advantage• Processes & guides are only useful if people use them! • Bringing about change, especially using evaluation, is
challenging – So:• Sustained vigilance – just keep going!• Have a vision while knowing that it's a long journey - keep
pushing ahead to achieve the vision • When the right conditions for change come along - ride on it
and run with it