James Herbert (Bachelor of Arts, Master of Criminal Justice)

EBP and GovernmentEBP and EvaluationDefinitionsEvaluation UseFrom Use to InfluenceResearch Questions & AimsMethodologyEthicsApplications

The Rising Prominence of Evaluation.What Does Evaluation

Actually Do?

“Evidence-based policy making is at the heart of being a reformist government... We cannot afford a Public Service culture where all you do is tell the Government what it wants to hear” – Prime Minister Kevin Rudd addressing the Australian Public Service Commission.

www.whitehouse.org- Restore Scientific Integrity to the White House: Restore the basic principle that government decisions should be based on the best-available, scientifically-valid evidence and not on ideological predispositions.

The Most Accessible Form of Evidence.The Most Common Form of

Evidence.Practical and Operationally

Oriented.

Very Applied ResearchPragmaticFocused on Operational IssuesLooks to Provide Direct

Benefit and Value

Dedicated Set of Resources and Activities Directed Towards Social ChangePrograms Can Become

Institutionalised Over Time

The Degree to Which the Program can Continue to FunctionFalls On a Continuum Between a

Program Ending and the Program Expanding into Other Jurisdictions

Instrumental Use Recommendations made by evaluators

are implemented Tends to not be direct, over long periods

of time with many indirect agents Not particularly common; politics tends

to supersede Example

Symbolic Use Evaluations are used to legitimise a

pre-empted course of action Does not include cases where

evaluation evidence is distorted, misused, or selectively disseminated

Example

Conceptual Use Individual evaluations do not provoke

a change, evidence accumulates and causes an enlightenment

The most powerful way by which evaluations can affect change

Imposed Use Legislative or regulatory requirement for

empirically tested programs United States Government Educational

Programs

Process Use The activity of engaging in evaluations

affects change separate to the outcome of the evaluation

Use has a long history in the study of evaluation, however: It lacks distinctiveness and firm definition It fails to recognise change processes at the

individual, interpersonal, and organisational levels

Use does not recognise the higher aim of evaluation as social betterment

Use suggests a directness and intentionality that rarely exists

Henry and Mark’s (2003) theory of change spans from individual, interpersonal, and organisational levels providing a comprehensive definition

The model enhances links to other more developed areas of research

Recognises the goal of evaluation as social betterment

Allows for the examination of influence through different levels

Can improve the validity of studies looking at the outcomes of evaluation

Allows for a more realistic assessment of what evaluation actually does

Henry & Marks (2003) provide a framework in which to understand change processes

Changes at different levels can act as a catalyst for further change

Examining ‘chains of influence’ may provide some insight into dynamic change processes

The narrative of the change D.A.R.E. Example

Social improvement is contingent on sustainabilityThe program sustainability

model enhances the evaluation influence model

Does evaluation influence how programs function? If so, how?Does evaluation influence affect

the sustainability of a social change program? If so, how?What program characteristics

affect evaluation influence?

To discover what evaluation does for programs and the management of public/private/philanthropic resources To find ways to improve the way

evaluation is used in order to increase the efficiency and effectiveness of program resources

Multiple Case Study Approach4-7 Social Change ProgramsAim: To Generate and Test

TheoryReplication Strategy Explanatory & Chronological

Analysis

Social Change Programs that Have Been Evaluated in the Last 12 Months Include cases with a diverse mix of

characteristics and sustainability factorsDevelop an extensive list of

potential programs to include

Program NameHost Organisation

Host Organisation Type

Program Description

Program TypeLocation

Evaluation Funding Source Website Contact

Outdoor Adventure Therapy

Taskforce NGO

Involves camps and physical activities for young people aimed at developing skills, confidence and pro-social interactions.

Delinquency Diversion. Adventure Therapy.

VIC

Website claims the program has been evaluated.

Donations, Bequests, Fundraising.

http://www.taskforce.org.au/ss_oat.htm

TaskForceHead OfficePO Box 369Prahran Vic. 31819521 4000enquiries@taskforce.org.au

Science and Technology Awareness Raising (STAR)

Murdoch University

University

University to school peer mentoring. Supporting science education. Academic support, positive role models and first hand advice.

Education. WAProfessional evaluation.

Ian Potter Foundation, BP, Department of Education and Training, Youth Affairs, Woodside, Social Ventures Australia, Rio Tinto WA Future Fund.

http://about.murdoch.edu.au/star/peer_tutoring/peertute.html

star@central.murdoch.edu.au

Midland Indigenous Youth Project

Swan Education District

Government

Academic and employment support for young Indigenous people in the Midlands area.

Education and employment support.

WA

Website gives comparative testing figures for outcomes.

Swan Education District and the Beacon Foundation with the support of the WA and Australian governments, local employers and Beacon sponsors, Newmont, Readymix and Sinclair Knight Merz.

http://www.beaconfoundation.net/other_projects.php

Tiesha McGilltiesha@beaconfoundation.net

Evaluation Characteristics Positive/Neutral/Negative Performance Internal/External Evaluation

Sustainability Performance Ceased/Reduced/Sustained/Expanded

Public Profile Popular or Community Support Exists/Neutral/Unpopular or

Community Unsupportive Program Type

Justice/Education/Health/Community Location

WA/VIC/NSW/QLD Host Organisation/ Funding Bodies

State Gov/Federal Gov/NGO/University/Private Companies

Protocol: What will be done in each case Interviews with program staff, evaluators, relevant

people within host organisation, relevant people within funding bodies

Analysis of key documents: Evaluation report, evaluation tender/scope documents

Triangulation of data Member Checking

Findings for each case: Evaluation influence (within program, host

organisation, funding body) Evaluation influence on sustainability of the

program Findings for across cases:

Possible common rules for how evaluation influence functions within programs

How different characteristics affect evaluation influence (program & evaluation characteristics)

Agreed terms and conditions between the programs, host organisations, funding bodies and the researcher

Consultation after data collection with participants and organisations involved

Voluntary participation Due discretion with sensitive materials

(contracts, tenders etc) Available counselling and contact details of relevant

organisational support (internal witnesses officer, equity officer)

Improved understanding of the evaluation process and the extent of its effects

Improved understanding of how evaluation is used in programs, different types of organisations, and funding bodies

Improved understanding of what program characteristics are associated with evaluation influence and how evaluation influence functions

Recommendations on how researchers can maximise the influence of their findings

Recommendations on how to manage the effects of evaluation