Measuring policy influence: like measuring thin air?

John Young: j.young@odi.org.uk Arnaldo Pellini: a.pellini.ra@odi.org.uk

RAPID

Power, Politics and evidence use

Evidence production and communication

Knowledge intermediaries and interactions

Policy processes are...

Monitoring  and  Evalua/on  

Agenda    Se3ng  

Decision  Making  

Policy  Implementa/on  

Policy  Formula/on  

Civil  Society  

Donors  Cabinet  

Parliament  

Ministries  

Private    Sector  

The Cynefn Framework

4

Policy change

•  Discursive: Client- focused services

•  Attitudinal: Farmers have good ideas

•  Procedural: Participatory approaches to service development

•  Content: UU20, UU25. New guidelines

•  Behavioural: Approach being applied in practice

5

Focus on behaviour change

Inputs Activities Outputs

Other Actors

Project Team

Outcome Impact Outcomes Impact Outcomes Impact Behaviour Change 6

RAPID Outcome Mapping Approach

Define your policy

objectives

RAPID Outcome Mapping Approach

Academic research communications

Develop a network or partnership

Online communications

Media strategy

More research

Policy advocacy coalition

Why do M&E?

•  To learn about what works •  To manage better •  To account:

– to donors – to recipients

9

Methods

•  Classical case studies (IDRC, IFPRI) •  Episode studies (ODI/RAPID) •  Stories of Change (Denning) •  Micro-Narratives (Snowden) •  Impact matrices (Davies) •  Peer evaluations (CHSRF) •  Systematic reviews? •  RCTs?

10

Outcome Mapping

OUTCOME MAPPING: Building Learning and Reflection into Development Programs Sarah Earl, Fred Carden, and Terry Smutylo

http://www.idrc.ca/en/ev-9330-201-1-DO_TOPIC.html

11

Social Network Analysis

www.odi.org.uk/RAPID/Tools/Toolkits/KM/Social_network_analysis.html 12

RAPID Outcome Assessment

www.odi.org.uk/RAPID/Publications/RAPID_WP_266.html 13

1.  Strategy and direction –are you doing the right thing?

2. Management –are you doing what you planned to do?

3. Outputs – are the outputs appropriate for the audience?

4.  Uptake – are people aware of your work?

5. Outcomes and impacts –are you having any impact?

A systematic approach

1.  Strategy and direction

2. Management

3. Outputs

4.  Uptake

5. Outcomes and impacts

14

Conclusions

Research to influence: •  Clear objectives

•  Understand the context

•  Theory of change

•  Iterative / learning approach

Measuring impact •  Clear objectives •  Theory of change •  5-levels •  Multiple methods •  Triangulation •  Expect the unexpected

15

Recommendations

•  Strategy: theory of change, impact pathways, peer review, log frame

•  Management: process records, appreciative inquiry, AARs, PRINCE2

•  Output: logs, peer-review,

•  Uptake: logs, webstats, surveys

•  Impact: outcome mapping, stories of change, episode studies, peer review

16

VDR 2010

VDR 2010: assessing policy influence

VDR 2010: Modern Institutions

•  Research output of a large governance programme (DfID)

•  12 months of research and consultations

•  Presented by WB + 13 donors in 12/2009

•  Policy influence assessment in 03/2011

VDR 2010

Confirm the policy change

Attitudinal change

Procedural change

Policy content

Behaviour change

Attitudinal change

Policy influencing approaches Evidence / science

based

Interest / values based

Inside track / cooperation

Outside track / pressure

Advising Advocacy

Lobbying Activism

Start and Hovland (2004)

VDR 2010

Areas of M&E

Strategy and direction Logframes, ToC

Management processes Project Governance, Research Management

Outputs Peer reviews, After Action Reviews

Uptake Citation Analysis, Uptake logs, User Surveys, …

Outcomes and impacts Stories of Change, Episode Studies, …

Hovland 2007

Strategy and direction Logframes, ToC

Management processes Project Governance, Research Management

Outputs Peer reviews, After Action Reviews

Uptake Citation Analysis, Uptake logs, User Surveys, …

Outcomes and impacts Stories of Change, Episode Studies, …

Areas of evaluation

Strategy and direction

Management processes

Outputs

Uptake

Outcomes and impacts

Planned

Data on uptake of VDR 2010

Stories of Change

Areas of M&E

What we did

•  Collaborate with a local research/advocacy organization

•  Citation analysis Google search

•  Interviews with donors representatives

•  Interviews with GoV advisers involved in the VDR 2010 process

•  Interviews with GoV officials

What we did/learned

•  Cultural norms influence the process

•  Citation, Google search, and time

•  Uptake and www traffic data

•  Management of the process emerged as an important theme

•  Perception of the quality of the report

•  Too early to speak of impact

Areas of evaluation

Strategy and direction

Management processes

Outputs

Uptake

Outcomes and impacts

Planned

Data on uptake of VDR 2010

Story of Change / Episode Study

What we did/learned

Perception about the timing of the VDR 2010

Perception of the process

Perception of the quality of the VDR 2010

Data on uptake of the VDR 2010

What we did/learned

Assessing the policy influence of research: a case study of governance research in Viet Nam ODI Background Notes, May 2012 (http://bit.ly/LNLHE0)

(http://bit.ly/LNOJZ1)

Assessment of the delivery of the Vietnam development report 2010 (Full report)

What we did/learned

•  ODI has 12 core research programs

•  145 publications / 74 blogs / 13 issues Development Policy Review and Disasters (2009-2010)

•  20 public events / 4500 participants (2009-2010)

•  Advisory work

M&E in ODI

Strategy and direction

ODI Strategy / Program Strategy / Reviews by SMT / ToCs

Management processes

Institute-wide management processes / staff surveys / project completion reports / after

action reviews / internal peer review

Outputs Peer reviews / quality assurance / communication team input / user surveys / end of project report

+ after action reviews

M&E in ODI

Uptake Note by communication team / CommStat / regular updates on

uptake / uptake log

M&E in ODI

CommStat ODI RAPID July – September 2011

M&E in ODI

M&E in ODI

Outcome / Impact

Stories of Change page / Review by TT heads (2008) / Program evaluations

M&E in ODI

http://www.odi.org.uk/work/stories-change/

M&E in ODI

Thank you