R&D Evaluation Methodology &
Funding Principles
Pilot testing
Conference in Prague, Oct 15, 2015
Vlastimil Růžička and KA4 team
www.metodika.reformy-msmt.cz
Pilot Testing („PT“) of the new
evaluation methodology
• Main objectives:
• comprehensively test the methodology including all its processes
• provide feedback for the future full-fledged evaluation
• Only a very limited number of RO selected, which perform exclusively or
partly research in chemistry and history
• Period for evaluation: 2010-14 (outputs 2009-13)
• Results of PT will not be published; however, results will be submitted to RO
for comments and feedback
• Results of PT will not influence the institutional funding for R&D
• The primary objective is not to evaluate the ROs, but to test the
methodology.1
Assessment criteria and sub-criteria
Assessment criteria Sub-criteria
I Research environment
The quality of the research management
(including HR management)
The adequacy of the research strategy
IIMembership of the global and national
research community
International research presence and
collaboration
National research presence and
collaboration
III Scientific research excellence
IV Overall research performance
Research output (including quantity and
overall quality)
Competitiveness in research
V Relevance for society
2
Pilot testing - timetable
3
XII/2014 I/2015 II/2015 III/2015 IV/2015 V/2015 VI/2015 VII/2015 VIII/2015 IX/2015 X/2015
invitation of ROs to the pilot testing
registration of RUs into the pilot testing
elaboration of the self-assessment report
searching for and contracting panel members
searching for and contracting reviewers
assessment of excellent outputs
calibration meeting of panel members14.,20.,21.5.
remote assessment of RUs
meeting of evaluation panels in Prague 29.6.-10.7.
elaboration of RU evaluation reports
elaboration of EvU evaluation reports
elaboration of panel members’ feedback to Metodika
elaboration of feedback to Metodika by evaluated units (RU, EvU)
List of participating EvUs; NS+ET
4
Name of EvU Type of RO
Nr. RUs in EvU
RUs registered in FoS
NA
TUR
AL
SCIE
NC
ES,
ENG
INEE
RIN
G
AN
D
TEC
HN
OLO
GY
University of Chemistry and Technology Prague - Faculty of Chemical Technology
ScRO-HEI 2 1.4, 2.5
University of Chemistry and Technology Prague - Faculty of Environmental Technology
ScRO-HEI 2 2.4, 2.7
University of Chemistry and Technology Prague - Faculty of Food and Biochemical Technology
ScRO-HEI 3 1.4, 1.6, 2.9
University of Chemistry and Technology Prague - Faculty of Chemical Engineering
ScRO-HEI 2 1.4, 2.4
Brno University of Technology -Faculty of Chemistry ScRO-HEI 4 1.4, 1.6, 2.5, 2.7
J. Heyrovský Institute of Physical Chemistry of the AS CR ScRO-ASCR 1 1.4
The Institute of Chemical Process Fundamentals of the AS CR ScRO-ASCR 4 1.4, 2.4, 2.5, 2.7
Centre for Organic Chemistry Ltd. IBRO- RTO 1 1.4
The Research Institute of Inorganic Chemistry, Inc. IBRO-RTO 1 1.4
List of participating EvUs; HU
5
• 12 RO, 15 EvU, 31 RU took part in pilot testing
• It was recommended to register RUs that correspond to the internal organizational structure of
RO as much as possible
• No interdisciplinary RUs registered even though RO were encouraged to do so
• The rule to register only one RU in the given FoS (to one subject panel) was adhered to
Name of EvU Type of RO
Nr. RUs in EvU
RUs registered in FoS
HU
MA
NIT
IES
The University of Pardubice - Faculty of Arts and Philosophy ScRO-HEI 3 6.1, 6.2, 6.3
The University of South Bohemia in České Budějovice - Faculty of Philosophy
ScRO-HEI 2 6.1, 6.2
The Institute of History of the AS CR ScRO-ASCR 1 6.1
The Institute for Contemporary History of the AS CR ScRO-ASCR 2 6.1, 6.3
The National Technical Museum NatRes 1 6.1
The National Archives NatRes 2 1.4, 6.1
Research Unit - recommendations
• For a registration of a RU a threshold has to be set, e.g. 50 outputs with a
possibility to increase the significance of „large“ outputs (e.g. SSH books)
• No limit to the maximum size of a RU (number of scientists and/or outputs)
• EvU has a possibility to register more than just one RU in a field of science if it
provides required documentation to justify the reasons (high number of
scientists and clear distinctiveness of sub-fields within the FoS)
• RU should clearly correspond to the organizational structure of the represented
EvU
• EvU does not necessarily need to include all scientific staff to the registered RU,
but it is obliged to inform the Panel about the reasons and the number of not
included scientists
6
Selection of panellists
• The process of searching and appointing the main and subject panel members as well as appointing the reviewers was provisional. It had to run much faster than what was required in the proposed cascade process.
• The selection of main and subject panel chairs, members of main and subject panels and the referees was done by KA4 team.
• Databases of experts who participated in analogous assessment of R&D in different countries was used. These databases included experts registered by Technopolis consortium, members of the main and subject panels in UK RAE2008 and UK REF2014 evaluations, experts from the evaluation in Portugal in 2013 organized by the Portuguese Foundation for Science and Technology, experts from the evaluation of the Royal University in Stockholm, experts from the University of Helsinki evaluations, experts used by the MEYS for the evaluation of its programs funding research.
• Some of the panel members were chosen based upon a reference of the evaluated units requested by KA4 team.
• In the last stage, recommendations from Technopolis Amsterdam for the missing experts in some disciplinary panels was used.
• Addressing and contracting potential experts was launched in January 2015 and finished in early June 2015.
7
List of panels and panellists
OECD Name Surname Name Affiliation, Town, Country
1. NATURAL SCIENCES- mainpanel
Thulstrup Erik Roskilde University, Roskilde, Denmark
Němeček Zdeněk Charles University, Prague, CR
Rejholec Václav Consultant for pharmaceutical industry, Prague, CR
1.4 Chemical Sciences
Hapiot Philippe CNRS- University of Rennes 1, Rennes , France
Guillon Daniel CNRS - University of Strassburg, Strassburg, France
Haines Michael Cofree Technology Ltd, Bricklehampton, UK
Heintz Andreas University of Rostock, Rostock, Germany
Kukhar Valery Nat. Academy of Sciences of Ukraine, Kyiv, Ukraine
Rizzi Andreas University of Vienna, Vienna, Austria
1.6 Biological Sciences
Driessen Arnold JM University of Groningen, Groningen, Netherlands
Elska Ganna National Academy of Sciences of Ukraine, Kyiv, Ukraine
Rodger Alison University of Warwick, Warwick, UK
8
• Main panel: chair-international expert, two Czech members representing research
community and users; third Czech member representing funding agency/ministry missing
• Subject panels: all international experts
• Number of subject panels members corresponded to the number of evaluated RU
List of panels and panellists
OECD Name Surname Name Affiliation, Town, Country
2. ENGINEERING and
TECHNOLOGY - main
panel
Seville Jonathan Peter Kyle University of Surrey, Guildford, Surrey, UK
Hanika Jiří Czech Academy of Sciences, Prague, CR
Souček Ivan University of Chemistry and Technology, Prague, CR
2.4 Chemical Engineering
Lapicque François CNRS-ENSIC University of Lorraine, France
Grievink Johan University of Technology, Delft, Netherlands
Ocone Raffaella Heriot-Watt University, Edinburgh, Scotland, UK
2.5 Materials Engineering
de With Gijsbertus Eidhoven University, Eidhoven, Netherlands
Drillon Marc CNRS - University of Strassburg, Strassburg, France
Katgerman Laurens University of Technology, Delft, Netherlands
Salmi Tapio Åbo Akademi, Åbo (Turku), Finland
2.7 Environmental
Engineering
Rulkens Wilhelmus Henricus Wageningen University, Wageningen, Netherlands
Legube Bernard University of Poitiers, Poitiers, France
Sánchez Hervás José María Unit for Energy Valor. of Fuels and Wastes, Madrid, Spain
2.9 Industrial
Biotechnology
Voragen Fons Wageningen UR University, Wageningen, Netherlands
Jelen Henryk Poznań University of Life Sciences, Poznań, Poland
9
List of panels and panellists
OECD Name Surname Name Affiliation, Town, Country
6. HUMANITIES - main
panel
North Michael University of Greifswald, Greifswald, Germany
Ledvinka Václav Prague City Archives, Prague, CR
Pešek Jiří Charles University, Prague, CR
6.1 History and
archaeology
Hadler Frank University Leipzig, Leipzig, Germany
Catalano Alessandro University of Padua, Padua, Italy
Hengerer Mark Ludwig-Maxmilians-Universität, Munich, Germany
Mayer Francoise Paul Valery University, Montpellier, France
Müller Leoš Stockholm University, Stockholm, Sweden
6.2 Languages and
literature
Achard-Bayle Guy University of Lorraine, Nancy, France
Balogh Andras Babeş-Bolyai-Universität, Cluj-Napoca, Romania
Raynaud Savina University Cattolica del Sacro Cuore, Milan, Italy
6.3 Philosophy, ethics
and religion
De Roover Jakob Ghent University, Ghent, Belgium
Müller Daniela University of Nijmegen, Nijmengen, Netherlands
Thomassen Einar University of Bergen, Bergen, Norway
10
Panellists
Panellists – country of residence
Country of residence Number Country of residence Number
Belgium 1 Norway 1
Czech Republic 6 Poland 1
Denmark 1 Austria 1
Finland 1 Romania 1
France 7 United Kingdom 4
Italy 2 Spain 1
Germany 4 Sweden 1
Netherlands 7 Ukraine 2
TOTAL 41
11
Average „success rate“ in contracting panel members: 124 addressed,
41 accepted, 33% success
Panels - recommendations
• Members of the panels are nominated by an institution responsible for evaluation
• Czech scientists will be full members of the subject panel, with their representation with a
maximum ratio of one Czech representative to two foreigners. Chairman of the panel is
always a foreigner, he/she will observe there are no conflicting issues stemming from the
presence of Czech panel members. The panel operates without advisers.
• The number of the panel members is related to the number of reviewed research units. This
is about 6 members for panels with up to 15 units, 9 members for 15 to 25 units, 12 members
when the number of units exceeds 25. The panel should not evaluate more than 40 research
units.
• In a remote assessment each unit is evaluated by three panellists (typically one Czech and
two foreigners), time for the assessment is from 0.5 to 1 day depending on the unit size.
• During the panel meeting the time specifically dedicated to the evaluation of one unit is
typically within 4 hours
• Number of panels can be expected between 30 and 35. For large scientific fields (e.g.,
physics or biology) two or three panels can be appointed, for small fields merging can be
introduced.
12
Self-assessment report („SAR“)
Submission Guidelines for the evaluated research organizations in pilot testing:
differences from the draft by Technopolis:
• No minimum number of participants (number of outputs / researchers)
• Introducing the organizational chart that specifies the context between EvU and RU
• Questions relating to the criterion I are on RU and EvU level or just on EvU level
• For a criterion of research excellence non-scholarly outputs were also accepted. There
is no strict condition for outputs in "English only„.
• A consistent form of HC tables for different types of RO
• Extending questions about post-gradual studies
• A brief description of the activity in the evaluated period is included
• A rationale for selection of EO-s required
13
Self-assessment report:
recommendations
• Structure: Narrative Section I, Data Section II
• Narrative: Summary - strategy, mission, goals (1 page); summary of past
activities over the evaluated period (3-4 pages), prospects for the next period (2-
3 pages)
• Data: blocks by 5 criteria, for the criterion I Environment RU will be presented in
the context of the EvU
• A list of all outputs linked with their authors will be presented, as well as a list of
researchers in the respective period (distinguishing those who are no longer
members during the evaluation or became members recently)
• Data to be normalized to FTE of researchers; need to establish clear rules to
calculate FTE researchers for Universities and for organizations whose
employees are engaged in science only partially
• The number of post-graduate students actually working in RU has to be clearly
stated14
Excellent outputs -
recommendations
• The number of outputs registered for evaluation will be specified,
outputs should not be restricted solely to the scientific field to which
the RU registered
• Non-scholarly excellent outputs should also be accepted, their
evaluation has to be specifically dealt with, descriptions in English have
to be provided to enable the evaluation by foreign reviewers
• If possible, outputs should be complemented with a bibliometric
analysis (WoS indicators, international review for SSH)
• Reviewers are selected by subject panels
• Two reviewers of the same level, while the third reviewer in case of a
difference of two or more points
• A new definition of "Grades", based on the definitions used in the pilot
testing
15
Bibliometrics -
recommendations
• Bibliometrics roughly as designed by TP, possibly can be simplified
• CZ Indicators: specify for the field where RU is registered and also
averaged by publishing a RU profile across disciplines
• Presentation of key indicators of bibliometric report also arranged in a
tabular form by RU ("matrix" form)
• Publications with authors from several RUs shall be considered for
each RU
• RIV expanded on book reviews
• Address the issue of taking into account the outcomes from applied
research (as done in pilot testing)
16
On-site visits
• They are not a part of the TP draft, mainly because of cost
• During SPE they were performed at all RUs; conclusion:
• The panels’ conclusive reflections were that the site visit was useful especially for
the institutions that under-presented their information in the self-assessment forms
or wrongly understood the information requirements. In the latter case, site visits
helped a lot in clarifying the aspects that RUs got wrong from the submission
guidelines. The panels suggested that in case site visits are absolutely out of the
question in the full-scale EM due to cost reasons, the evaluation agency may
consider organizing a Q&A session using online tools and/or videoconferencing.
(see Background report 10: The Small Pilot Evaluation – Feedback and Results, part 2.2.1)
• The prevailing opinion of the panellists during PT:
• visits are important or even indispensable
• „Hearing“: cheaper and in terms of organization easier option: visits of RU
representatives at the site of panel meeting
17
On-site vists - recommendations
• On-site visits – non-uniform positions within the IPN Metodika team as
well as among panellists and RUs
• the problem mainly in timing and logistics, and also in cost
• panellists mostly pro - personal contact between evaluators and
evaluated ones is desirable
• hearing as a compromise solution,
• require a firmly set agenda
18
Calibration
1st Interim Report (1IR):
• main panel chairs to ensure coherence in the interpretation of the assessment criteria and their key words
among the panels, so that, for example, a score 4 against the criterion ‘research performance’ has an
equal value in physics as in social sciences, for any type of organization. (1IR, part 4.8.1, page 80)
• Subject panels will undertake early calibration exercises to ensure that subject panel members (and
referees) develop a common understanding of the quality levels. The subject panels will continue to discuss
the application of the quality levels and will keep under review the scoring patterns of panel members
throughout the assessment process to ensure consistency in the subject panel’s standards of assessment.
(1IR, part 5.1.3, page 89)
PT:
• The only calibration meeting: the chairs and members of the main panel and the chairs / vice-chairs of
subject panels (May, Prague)
• Inadequate calibration is a weakness of PT: e.g. significantly different quality grades between different
subject panels
• Recommendations mentioned in the documents drawn up by Technopolis consortium should be expanded
and clarified
19
Calibration - recommendations
• Precisely define the number, the agenda and the participants of the
calibration meetings.
• In calibration meetings define the role and tasks of the main panel chair, and
the chairs of the subject panels and in implementing conclusions of these
meetings.
• Accurately describe the procedure that will guarantee the panels had a
uniform interpretation of the quality degrees (grades) in the evaluation by all
five criteria. Uniform interpretation must be ensured between the subject
panels within one disciplinary area as well as between different main
panels.
• Reassess the need for field-specific terms as suggested by Technopolis:
• significance, originality, rigour in criterion III "Scientific excellence"
• reach, significance in the criterion in the "Societal relevance"
• Reassess the need for the weight settings of sub-criteria in the criteria I, II,
and IV.
20
The evaluation report for EvU
• TP proposal is methodologically weak: „The Subject Panel Chair for the major field of
involvement of an EvU will be responsible for the analytical report at the EvU level, based upon a
draft developed by the Evaluation Management Team and in cooperation with the other relevant
Subject Panel Chairs“ (Summary report, part 3.2.4)
• TP template for the evaluation report for EvU has a form of a SWOT analysis
• A number of panelists expressed their doubts about the benefits of the
evaluation report for EvU; in addition, from self-assessment report it has not
been possible to obtain sufficient information for its creation
• At the panel meetings in Prague, it was agreed that EvU report will be
prepared by the chair of the main panel in collaboration with the chairs of
subject panels
21
EvU evaluation report -
recommendation
• It has to be clarified whether in a selected field-defined RU evaluation
methodology preparing of a report on the level EvU is possible and
necessary
• It is not clear how the report should look like and who will be in charge
of preparing it
22
Feedback from panellists
• Some comments already included in the above recommendations
• Similar regular evaluations of R&D are common in some European
countries (e.g. UK, FR, PT, IT, LT, LA)
• SSH:
• METODIKA introduces clear rules
• peer-review by foreign experts brings positive experience, limits potential bias of
national experts
• specific requirements for bibliometric report
• Definition of RU is somewhat questionable as RU goes across
organizational structure of EvU (exact sci.)
• There should be some way of checking on the truth content of what has
been written (in SAR)
• A clear idea of future vision and strategy to achieve it must be provided
23
Feedback from RUs/EvUs
• RUs consider present evaluation method (Metodika2013+) better,
simpler, more accurate, more objective (mostly exact sci., also some
humanities)
• RUs critical to the structure of the SAR (exact sci.), make it shorter
• 1-2 % excellent outputs is too little
• RUs critical to the bibliometric report (both exact sci. and humanities)
• Evaluation panel made a realistic assessment only after the site visit
• Out of 5 RUs in exact sci., 3 critical to site visit; in humanities mostly
positive
• Adequacy of the evaluation can be judged only when a certain minimum
number of RUs is taking part in
• Majority of RUs with panel´s conclusions/recommendations
24
Conclusions
• Evaluation based on methodology by Technopolis is feasible
• excerpts from feedback by panellists:
• performing a self-evaluation to be read by experts in itself can be a highly useful
exercise for an institution as well as for the individual researchers
• encourages institutions to assess their own performance against excellence defined on a
global scale. A result of this could be a better understanding by them of their competitive
positioning and could positively influence their goal setting
• Some modifications are needed, based also on experience from the on-
going evaluation by ASCR and on experience from METODIKA2013+
• Next evaluation should involve a smaller segment of RO-s rather than
all RO-s in the Czech Republic. Evaluation of all Czech universities is an
option
25
Thank you for your attention
www.metodika.reformy-msmt.cz