evaluating the impact of health research: revisiting the canadian institutes of health research...
TRANSCRIPT
Evaluating the impact of health research:
Revisiting the Canadian Institutes of Health Research Impact Assessment Framework
Nicola Lauzon, Marc Turcotte & Laura McAuley
Impact Assessments Unit
Canadian Institutes of Health Research
Ottawa, Ontario, Canada
Presentation Overview
• CIHR impact assessment framework history
• Study purpose & methods
• Findings• Uses of CIHR framework• Focus group feedback from framework users
• Application of Lessons learned• Changes to the framework categories• Development of a simplified conceptual model
• Next steps
• Questions
2
CIHR Impact Assessment Framework 2006
Category Description
Advancing Knowledge
• discoveries, breakthroughs, and contributions to the scientific literature
Building Capacity
• development and enhancement of research skills in individuals and teams
Informing Decision-Making
• impacts in the areas of science, public, clinical, and managerial decision-making, practice and policy
Health & Health Systems Impacts
• advances in prevention, diagnosis, treatment and palliation, functioning of the health system
Economic Impacts
• commercialization of research discoveries and human capital gains
3
Study Purpose & Methods
Purpose• To document lessons learned and inform framework improvement
Methods• Document review of impact reports & protocols
• How has it been used, results of use, lessons learned
• Focus groups• Feedback and input from users and potential users of framework
• Six groups; total 35 participants
4
Results: Document Review
Uses of CIHR framework
• Topic-based assessment• Focus on specific disease condition
• Investigate impacts across multiple categories
• Institute-based assessment• Focus on research mandate areas of CIHR Institutes
• Investigate impacts across multiple categories
• Category-based assessment• Focus exclusively on one impact category
5
Topic-based impact assessments
Topic• Cardiovascular disease• Obesity• SARS
Main Lessons learned• Validation of CIHR impact categories• Impacts continue after 20+ years• Attribution difficult• Topic dictates types of impacts seen
6
Institute-based impact assessments
CIHR International Review (2010 – 2011)
• Impacts of broad mandate areas of all 13 CIHR Institutes between 2000 – 2010
Lessons learned• Limited availability of indicators in latter impact categories• Need to revisit the categorization and utility of certain indicators• Potential to improve bibliometric methodology
7
Results: Focus Groups
Main theme of discussions:• Attribution versus contribution
• Direct attribution desired
• Struggle to find value of latter categories and associated indicators
Lessons Learned• Need to clarify to stakeholders:
• Our conceptualization of the research impact process
• How CIHR fits into the research impact process
• How the CIHR framework fits into the research impact process
8
Simplified conceptual model of research impact
9
Research Enterprise Research Users
Society
KnowledgeDecision-Making
Health
Knowledge Translation
Research Findings EffectUseFunding
Impacts on…
Capacity
Health System
Economy
Research Enterprise Research Users
Society
KnowledgeDecision-Making
Health
Knowledge Translation
Research Findings EffectUseFunding
Impacts on…
Capacity
Health System
Economy
Not-so-simplified version
10
Research Enterprise
Research Users
Society
KnowledgeDecision-Making
Health
Knowledge Translation
Research Findings EffectUseFunding
Impacts on…
Capacity
Health System
Economy
Not-so-simplified version
11
Government priorities
Public prioriti
es
Funder priorities
Gov
ernm
ent b
udge
t
Partnerships &
leveraged funds
Pe
er re
view
Researcher
priorities
Research
Institution
priorities
Infr
astr
uctu
re
Re
sea
rch
te
am
co
mp
osi
tion
Publicatio
n bias
Pe
er re
view
Re
sea
rch
er
cha
ract
eris
tics
Tim
e la
g t
o p
ub
lica
tion
Continued relevance
User
characteristics
Governm
ent
prioritiesU
ser a
ccep
tan
ce
Acce
ssibility
Sys
tem
cap
acity
Eco
nom
ic o
utlo
ok
Per
sona
l ch
arac
teris
tics
Dem
and
Po
litical clim
ate
Syste
m ca
pa
city
Simplified conceptual model of research impact
12
Research Enterprise Research Users
Society
KnowledgeDecision-Making
Health
Knowledge Translation
Research Findings EffectUseFunding
Impacts on…
Capacity
Health System
Economy
Simplified conceptual model of research impact
13
Research Enterprise Research Users
Society
KnowledgeDecision-Making
Health
Knowledge Translation
Research Findings EffectUseFunding
Impacts on…
Capacity
Health System
Economy
Simplified conceptual model of research impact
14
Research Enterprise Research Users
Society
KnowledgeDecision-Making
Health
Knowledge Translation
Research Findings EffectUseFunding
Impacts on…
Capacity
Health System
Economy
Simplified conceptual model of research impact
15
Impacts on…Knowledge
Capacity
Decision-Making
Health
Health System
Economy
Research Enterprise Research Users
Society
Knowledge Translation
Research Findings EffectUseFunding
16
CIHR Impact Assessment Framework
Category Lower order indicators Higher order indicators
Knowledge • # Publications from grants in specific research area
• # Canadian publications in specific research area
Capacity • # CIHR-supported PhD trainee awards
• # Canadian PhD grads in health
Decision-Making
• Citation in research • Citation in health policy
Health • Outcome measures of condition X for clinical trial patients
• Prevalence of condition X in Canada
Health Systems
• Health efficiencies from specific grants
• Wait times
Economy • License or royalty agreement counts
• # jobs created from spin-off companies
Next steps & Conclusions
• Continue testing model, framework categories and indicators
• Feedback & validation from stakeholders
• Alignment of model and framework with literature
17