european trends in the use of evaluations and indicators for performance-based funding of research...
TRANSCRIPT
European Trends in the Use of Evaluations and Indicators for Performance-Based Funding of Research Institutions
Gunnar Sivertsen
Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway
Overview of this presentation
1. The dual funding system in Europe
2. Research evaluation and performance indicators
3. Evaluation-based versus indicator-based institutional funding
4. New trends:
a. Evaluation of societal impact
b. Data from Current Research Information Systems
5. Conclusions
Overview of this presentation
1. The dual funding system in Europe
2. Research evaluation and performance indicators
3. Evaluation-based versus indicator-based institutional funding
4. New trends:
a. Evaluation of societal impact
b. Data from Current Research Information Systems
5. Conclusions
”The dual funding system” (Public sources)
Government
Direct grants
Historical, Political
Strategic
Performance based
Research institutions
Research Councils
Projects and Programmes
Competition
Research Evaluation
The balance?
Government
Direct grants
Historical, Political
Strategic
Performance based
Research institutions
Research Councils
Projects and Programmes
Competition
Research Evaluation
The mix?
Government
Direct grants
Historical, Political
Strategic
Performance based
Research institutions
Research Councils
Projects and Programmes
Competition
Research Evaluation
Same aims, methods, and effects?
Government
Direct grants
Historical, Political
Strategic
Performance based
Research institutions
Research Councils
Projects and Programmes
Competition
Research Evaluation
Overview of this presentation
1. The dual funding system in Europe
2. Research evaluation and performance indicators
3. Evaluation-based versus indicator-based institutional funding
4. New trends:
a. Evaluation of societal impact
b. Data from Current Research Information Systems
5. Conclusions
Research Evaluation and Performance Indicators
Bibliometrics may inform, but not replace peer review
A. Research Evaluation without funding implications
Bibliometrics may inform, but not replace peer review
Bibliometrics informs the evaluation. Evaluation results are not translated into funding
B. Research Evaluation with funding implications (PBF)
Bibliometrics may inform, but not replace peer review
Bibliometrics informs the evaluation. Evaluation results are translated into funding
C. Performance-based funding (PBF) without Research Evaluation
Bibliometrics is translated into funding. Often combined with other indicators.
Overview of this presentation
1. The dual funding system in Europe
2. Research evaluation and performance indicators
3. Evaluation-based versus indicator-based institutional funding
4. New trends:
a. Evaluation of societal impact
b. Data from Current Research Information Systems
5. Conclusions
Examples of C: Indicator-based models
Bibliometrics is
translated into funding.
Often combined with other indicators.
1990
Examples of B: Evaluation-based models
2010
Bibliometrics informs the evaluation. Evaluation results are translated into funding
1986
2003
2018?
2018?
Examples of B and C: Combinations in practice
2010
Bibliometrics is translated into evaluation. Evaluation results are translated into funding
2003
Examples of A: Evaluation without funding implications
2010
Bibliometrics informs the evaluation. Evaluation results are not translated into funding
Overview of this presentation
1. The dual funding system in Europe
2. Research evaluation and performance indicators
3. Evaluation-based versus indicator-based institutional funding
4. New trends:
a. Evaluation of societal impact
b. Data from Current Research Information Systems
5. Conclusions
The conclusion after an open consultation in UK in 2006: Is it possible to use metrics instead or as a supplement?
Bibliometrics (citation indicators) may inform, but not replace peer review
The conclusion after an open consultation in UK in 2006: Is it possible to use metrics instead or as a supplement?
We are not covered!!
2018?
2014
• Evaluation panels may be provided
with bibliometrics upon request.
• Use of journal impact factors and
other bibliometric indicators not supplied through the REF
administration is forbidden.
Impact template as Word file (6,975 submissions)
“For the impact component of the REF, it is not currently feasible to use quantitative indicators in place of narrative impact case studies, or the impact template.”
Overview of this presentation
1. The dual funding system in Europe
2. Research evaluation and performance indicators
3. Evaluation-based versus indicator-based institutional funding
4. New trends:
a. Evaluation of societal impact
b. Data from Current Research Information Systems
5. Conclusions
Indicator-based models are increasingly based on institutional or national Current Research Information Systems (CRIS)
Principles behind the use of institutional data in a shared national Current Research Information System • Completeness: All scholarly
publications and other results from research are included
• Transparency: Every institution can see and check all other institutions’ data. The national database is also online and open to society at large.
• Multiple use of the data: CV’s, applications, evaluations, annual reports, internal administration, bibliography for Open Archives, links to full text, etc.
Indexed journals
Journals and series
Books
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Health
Sciences
Natural
Sciences
Engineering Social Sciences Humanities
Coverage of 70,500 scholarly publications from the
higher education sector in Norway 2005-2012.
ScopusWeb of Science
Coverage of scholarly publications in Norway’s CRIS (100 per cent) compared to coverage in WoS and Scopus
Both Elsevier and Thomson Reuters are now integrating the citation databases with research management tools and CRIS-systems
Overview of this presentation
1. The dual funding system in Europe
2. Research evaluation and performance indicators
3. Evaluation-based versus indicator-based institutional funding
4. New trends:
a. Evaluation of societal impact
b. Data from Current Research Information Systems
5. Conclusions
Conclusions: European Trends in the Use of Evaluations and Indicators for
Performance-Based Funding of Research Institutions
Both evaluation-based and indicator-based solutions prevail
PBF is effective even if introduced as a small share of the total funding. The discussion is about positive versus negative effects
Difficult to identify “best practice” or overall positive/negative effects
Increasingly important:
Evaluation of societal impact
Use of CRIS systems