real world approaches to assessing the impact of research on policy annette boaz kings college...
TRANSCRIPT
![Page 1: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/1.jpg)
Real world approaches to assessing the impact of research on policy
Annette Boaz
King’s College London
23rd September 2010
![Page 2: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/2.jpg)
2
Outline
• Models of the relationship between research
policy and practice
• Rationales for assessing impact
• What can we learn from the literature?
• 10 research impact assessment case
studies
• Consider the implications for SSH research
![Page 3: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/3.jpg)
3
Models
Linear – relay race
Relationship – two communities
Systems – co-production
Diffusion
Engineering
![Page 4: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/4.jpg)
4
Activities designed to promote research
impactKnowledge
brokers
Research briefings
Participation in Networks
Co-locating researchers and
policy makersJoint
research projects
Advisory groups
Seminars/ presentations
![Page 5: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/5.jpg)
5
Models
Linear – relay race
Relationship – two communities
Systems – co-production
Research briefings
Joint research projects
Advisory groups
![Page 6: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/6.jpg)
Effective knowledge transferdepends on
• the nature of the research and its utility
• the personal characteristics of both researchers and users
• the links between researchers and users
• the context for the application of the research
See Nutley, Walter and Davies, 2007, pp 81-83,
See also Overseas Development Institute
6
![Page 7: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/7.jpg)
7
Why assess the impact of our research?
Learning
Performance
In its quest for excellence, the European Commission must and will encourage, promoteand support every effort to understand and monitor the quality of research at universities. 1
Accountability
Promoting organizational achievements
Moderating between stakeholders
1Commissioner Janez Potočnik, Assessing Europe’s University-Based Research, 2010
![Page 8: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/8.jpg)
8
Review question: How do you (best) evaluate the impact of research on
policy?• Explored the websites of 30 organizations
• Hand search of selected journals and reference lists
• Search of SPRU and IDS library catalogues
• Search of 10 research databases
• 1987-2007
• UK and international sources
![Page 9: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/9.jpg)
Research programmes
Research
Initiatives to increase research use
Research Users
Systematic reviews
Focusing the review on policy use
Single study
Policy makers
![Page 10: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/10.jpg)
Country of OriginAustralia, 8
Canada , 20
Finland, 4
Switzerland, 3
The Netherlands, 9
UK, 61
USA, 38
Other,13
Other includes: Brazil, Denmark, Spain and Norway
![Page 11: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/11.jpg)
11
Methods for measuring research impact
Figure 5: Methodologies
0
10
20
30
40
50
60
To
tal n
um
ber
of
pap
ers
Discussed
Used
![Page 12: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/12.jpg)
How do people evaluate the impact of research
programmes? Wide range of methods and approaches (often in combination):
· Qualitative methods: semi-structured interviews, documentary analysis, field visits and observations
· Quantitative methods: surveys, bibliometrics and patent/new technology tracking
· Panels and peer review
· Workshops and focus groups
· Process tracking
![Page 13: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/13.jpg)
Figure 6: Frameworks for structuring and interpreting data
02468
101214161820
Tota
l nu
mb
er
of
pa
pe
rs
Discussed
Used
![Page 14: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/14.jpg)
Ten impact assessment case studies
Stockholm Environment Institute
14
The Foundation for Research, Science and Technology (New Zealand)
Finnish Environment Institute
The Irish Environmental Protection Agency
CGIAR
Environment Agency (England and Wales)Defra
Land and Water Australia
French Ministry of Ecology, Energy, Sustainable Development and the Sea
Swedish Environmental Protection Agency
![Page 15: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/15.jpg)
15
The Irish Environmental Protection Agency
![Page 16: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/16.jpg)
Finnish Environmental Institute
Embedded approach: Ten year independent evaluation, four year strategic evaluations, internal annual evaluations, annual researcher self evaluation
Goal: how to increase impact
Methods: multi-method16
![Page 17: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/17.jpg)
Key themes:
• Diverse of objectives, scope and scale
• Types of approach
• Generic challenges
• Challenges specific to environmental research
17
![Page 18: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/18.jpg)
What are the implications for SSH
research in Latin America and Europe?
18
![Page 19: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/19.jpg)
8 key questions (when designing an impact
evaluation)1. What is your conceptual framework?2. What are the outcomes of interest?3. What methods will best explore the outcomes of interest?4. How do you address attribution?5. What is the direction of travel for the evaluation?6. Is this a mixed method approach, providing scope for triangulation?7. Will the methods selected capture context and the complexity?8. When might be the best time to conduct the evaluation?
These need to be considered in the light of the time, skills and resources available for the evaluation.See Boaz et al 2009
![Page 20: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/20.jpg)
My contact details: [email protected]
Some useful references and websites:
-Boaz, A, Fitzpatrick, S and Shaw, B (2009) Assessing the impact of research on policy: a literature review. Science and Public Policy, 36(4): 255-270
-Nutley S, Walter I, and Davies H (2007) Using evidence: How research can inform public services. Bristol: The Policy Press.
-O’Leary, G, Boyle, R, Donlon, B and Sheils, L (2008) Providing a framework for accountability and learning in environmental research. Administration. 5 (4): 159-170
-The Overseas Development Institute RAPID programme: http://www.odi.org.uk/programmes/rapid/
- Wooding S, Nason E, Klautzer L, Rubin J, Hanney S and Grant J (2007) Policy and practice impacts of research funded by the ESRC. A case study of the Future of Work programme, approach and analysis. Cambridge: RAND Europe. http://www.rand.org/pubs/technical_reports/2007/RAND_TR435.pdf
20
![Page 21: Real world approaches to assessing the impact of research on policy Annette Boaz Kings College London 23 rd September 2010](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5513c40255034679748b46b8/html5/thumbnails/21.jpg)
21
Managing Editors: Annette Boaz, Division of Health and Social Care Research, King's College London, UK and David Gough, Institute of Education, University of London, UK
Volume 6, 2010, 4 issues: January, May, August and November http://www.policypress.co.uk/journals_eap.asp