www.3ieimpact.org philip davies making evidence accessible and relevant for policy and practice...

www.3ieimpact.orgPhilip Davies

Making Evidence Accessible and Relevant for Policy and Practice

Philip Davies

International Initiative for Impact Evaluation [3ie]

Africa Evidence Network Regional MeetingJohannesburg, South Africa, 3rd June 2015

www.3ieimpact.orgPhilip Davies

• Helping people make better decisions and achieve better outcomes, by using the best available evidence from research and other sources

• Knowing what are effective interventions (“what works?”)

• In achieving which outcomes?

• For which groups of people?

• Under what conditions?

• Over what time span?

• At what costs?, plus

• Integrating research with decision makers’ knowledge, skills, experience, expertise and judgement

What is Evidence-Based Policy

www.3ieimpact.orgPhilip Davies

Evidence

Experience & Expertise

Judgement

Resources

Values, Beliefs and

Ideology

Habits & Bureaucratic

Culture

Lobbyists & Pressure Groups

Pragmatics & Contingencies

Factors Other Than Evidence

www.3ieimpact.orgPhilip Davies

Understanding

the Problem(Conceptualisation

) Developing

Solutions

(Policy Develop

ment)Putting Solution

s Into Effect

(Implementation)

Monitoring and

Evaluation

(M&E)

The ‘Classic’ Policy Cycle The ‘ROAMEF’ Policy Cycle

Evidence is required across the entire policy cycle

www.3ieimpact.orgPhilip Davies

The Kind Of Evidence Decision-Makers Look For

• Identifying the nature, size and dynamics of the problem

• Specifying the desired objectives• Identifying viable policy options• Identifying how the policy is supposed to work• Identifying the likely and achieved outcomes/impacts• Identifying the social distribution of

outcomes/impacts• Understanding people’s attitudes, experiences,

behaviour• Valuing the impacts (cost-benefit/cost-effectiveness)• Identifying effective implementation and delivery

www.3ieimpact.orgPhilip Davies

Types of Evidence for Policy Making

www.3ieimpact.orgPhilip Davies

Theory of Change/Logic Model/Programme Theory

• How is a policy/programme supposed to work?

• What activities, mechanisms, people, outputs have to be in place?

• And in what sequence – what is the causal chain?

• What resources are required – and are available?

• What data are required – and are available?

• Is the policy/programme feasible/achievable?

www.3ieimpact.orgPhilip Davies

Constituent Features of a Theory of Change

Assumptions?

www.3ieimpact.orgPhilip Davies

Constituent Features of a Theory of ChangeData Required

•Surveys, statistics, demographic data

•Qualitative data•Costs/benefits data•Systematic review data•Documentary analysis

• Performance data• Historical data• Diversity data• Qualitative data• Effectiveness data

• Stakeholder data• Qualitative data• Public opinion data• Effectiveness data

• Performance data• Effectiveness data• Stakeholder data• Qualitative data• Costs/benefits data

• Administrative data• Performance data• Costs/benefits data

• Administrative data• Performance data• Qualitative data

• Counterfactual data• Administrative data• Survey data, statistics• Cost/benefit data

www.3ieimpact.orgPhilip Davies

Types of Systematic Review

• Statistical Meta-Analyses

• Narrative Systematic Reviews

• Qualitative Systematic Reviews

• Rapid Evidence Assessments

• Evidence Maps and Gap Maps

www.3ieimpact.orgPhilip Davies

Single studies can:

Misrepresent the balance of research evidence

Illuminate only one part of a policy issue

Be sample-specific, time-specific, context-specific

Often be of poor quality

Why Do We Need Systematic Reviews?

Consequently, give a biased view of the overall evidence

www.3ieimpact.orgPhilip Davies

• Systematic searching for studies

• Systematic critical appraisal of identified studies – separating the wheat from the chaff

• Systematic and transparent inclusion/exclusion of studies for final review

• Systematic and transparent extraction of data

• Systematic statistical testing and analysis

• Systematic reporting of findings

What Makes a Review Systematic?

www.3ieimpact.orgPhilip Davies

Statistical Meta-Analytical Reviews

Source: David B. Wilson, 2006, A systematic review of drug court effects on recidivism

www.3ieimpact.orgPhilip Davies

• Synthesise qualitative and ethnographic evidence

• In-depth interviews, focus groups, observational studies, documentary analysis, case studies

• Seek common themes, concepts and principles across different studies

• Detailed attention to context/contextual specificity

• And stakeholders’ views

• Do not seek generalisations

Qualitative Systematic Reviews

Photo © Albert Gonzalez Farran - UNAMID

www.3ieimpact.orgPhilip Davies

• Scaled down systematic reviews of existing evidence

• Timed to meet the needs of policy makers/practitioners (1-3 months)

• Strategically using the ‘three arms’ of systematic searching, but less exhaustively

Rapid Evidence Assessments – What Are They?

• Critical appraisal of identified studies is included

• Summary of findings, with caveats and qualifications

Photo © Panos East Africa

www.3ieimpact.orgPhilip Davies

Rapid Evidence Assessments – How Scaled Down?

www.3ieimpact.orgPhilip Davies

3ie Evidence Gap Maps

• Maps of the existing evidence base on a policy issue, topic or sector such as maternal health, HIV/AIDS, agriculture, extreme poverty

• Structured around a framework of interventions and outcomes (intermediate and final)

• A ways of identifying where there is evidence, and where there is not

• An indication of the quality of this evidence

• Links to user-friendly summaries in the 3ie database of systematic reviews.

www.3ieimpact.orgPhilip Davies

Rapid Evidence Assessments - LimitationsEvidence Gap Maps

www.3ieimpact.orgPhilip Davies

www.3ieimpact.orgPhilip Davies

www.3ieimpact.orgPhilip Davies

Some Key Sources of Sythesised Evidence

• 3ie Impact Evaluations Database(http://www.3ieimpact.org/en/evidence/impact-evaluations/)

• 3ie Systematic Reviews Database • (http://www.3ieimpact.org/en/evidence/systematic-reviews/)

(http://www.3ieimpact.org/en/evidence/systematic-reviews/)• Best Evidence Encyclopedia (http://www.bestevidence.org/)• Cochrane Collaboration (www.cochrane.org)• Campbell Collaboration (www.campbellcollaboration.org)• Collaboration for Environmental Evaluation (http://www.environmentalevidence.org/)• National Institute for Health and Clinical Excellence (http: www.nice.org.uk/)• NHS Evidence (http://www.evidence.nhs.uk/)• National Guidelines Clearinghouse (USA) (www.guidelines.gov)• Prospero: International Prospective Register of Systematic Reviews (http://www.crd.york.ac.uk/prospero/• Social Care Institute for Excellence (http://www.scie.org.uk/)• Social Programs That Work (http://evidencebasedprograms.org/)

www.3ieimpact.orgPhilip Davies

UK Policymakers’ Views of Evidence

• Focus on the ‘end product’, rather than how the information was either collected or analysed

• Use of ‘anecdotal’ evidence (“tells a story”)• Drawing on such things as ‘real life stories’,

‘fingers in the wind’, ‘local’ and ‘bottom-up’ evidence

But:

• “If we try and move anywhere without having the scientific basis to do so we get fleeced in the House”

• And: DfID Evidence into Action Team

• And: BCURE Programme + DPME (South Africa)

www.3ieimpact.orgPhilip Davies

Sharks

Where Do UK Civil Servants Go For Evidence?

PlanktonAcademic/Evaluation Research?

www.3ieimpact.orgPhilip Davies

UK Policymakers’ Views of Research Evidence

• Too Long • Verbose• Too Detailed • Too Dense• Impenetrable• Too Much Jargon• Too Methodological• Untimely• Irrelevant for policy

Source: Campbell, S., et al; 2007, Analysis for Policy

www.3ieimpact.orgPhilip Davies

• Establish what research says and does not say

• Establish the policy messages and policy implications

• Use a 1:3:25 format

• Very little mention of methodology

• Be clear - plain English summary

• Be persistent and opportunistic

Improving Communication of Evidence

www.3ieimpact.orgPhilip Davies

Thank you

Philip Davies

Email: pdavies@3ieimpact.org

+44 (0)207 958 8350

Visit www.3ieimpact.org