Workshop on research assessment in CERIF

Stephen Grace, Brigitte Jörg, Aija Kaitera, Maximilian Stempfhuber

2

Workshop on research assessment in CERIF

• Three ten-minute presentations on systems in UK, Germany and Finland

• One hour discussion on common data elements across different national boundaries

• Map them to CERIF

• Submit to CERIF Task Force

Questions to ask

• WHO does it? Is it local or national?

• WHEN and WHERE it takes place? One-off or regular activity?

• WHY is it done? What is the purpose of the exercise – to allocate money, to improve performance, to encourage co-operation etc.?

• WHAT information/data is needed? The types of information needed?

• HOW universities handle the process? Systems, staff involved?

3

4

Assessment in the UK Stephen Grace

Research Assessment Exercise (RAE):•WHO does it? Funding Councils•WHEN? Every six years•WHERE? From University to Funding Council•WHY? To identify Research Excellence in order to

allocate funding•WHAT? Information on Researchers, their Outputs,

Sources of Funding, Studentships, Measures of Esteem•HOW? Gather data from internal systems to submit to

the Funding Council using XML Schema or online form filling

5

Research Excellence Framework (REF)

• Research Excellence Framework (REF) aims to be simpler to administer than RAE

• Introduces Impact element (25% of marks) to cover benefits to economy, society, culture, public policy and services, health, environment, international development and the quality of life

• Still working out what will be asked, and the form of

submissions in the area of Impact

6

Research Rating:

• WHO does it? German Research Council (Wissenschaftsrat)

• WHEN? Since 2007, pilot phase for selected disciplines

• WHERE? At Universities and research centres jointly funded by federal ministries and states

• WHY? To rate and compare research performance

• WHAT? Information on Research Groups: Quality, Impact / Effectiveness, Promotion of young Scientists, Transfer

• HOW? Questionnaire for all information except publications; self reporting system for publications or use of commercial reference database (depending on discipline)

Assessment in Germany Maximilian Stempfhuber

7

Assessment in Germany Maximilian Stempfhuber

• Rating (not ranking)

• Pilot studies in Chemistry and Sociology (2007/08), different modes of data collection

• Current: Electrical engineering & Information technology, planned: Humanities

• Currently not used for performance based funding

• In parallel:• University Excellence Initiative (Wissenschaftsrat & DFG)• CHE University Ranking (CHE & Die Zeit)

1: Quality of Research

• Citations• Per selected publication• Per selected publication, normalized to journal

• No. of reviewed publications

• Publications• List of publications per publication type• 5 most important publications• List of patents

1: Quality of Research (cont.)

• Total of external funding spent• Public funding• Commercial funding

• List of research awards etc.

• Management of national / international research networks

2. Impact / Effectiveness

• Publications: No. of reviewed publications

• Patents: No. of patents, total of earned licensing fees

• Total of external funding spent• Public funding• Commercial funding

• Percentage of external funded staff

• Management of national / international research networks

2. Impact / Effectiveness (cont.)

• Citations of selected (important) publications

• Citations of selected (important) publications normalized to journal

• No. of funded visiting researchers

• No. of distinguished positions in external scientific organizations, learned societies etc.

• Chair / co-chair at conferences

• Ratio: reviewed publications / scientific staff

3. Promotion of young scientists• PhDs

• No. of successful PhD studies / female applicants

• List of PhD (structured) programmes• No. of advised and finished PhD thesis

• No. of persons receiving grants

• List of first-time professorships

• No. of habilitations / female applicants

4. Knowledge / Technology Transfer

• No. of patents, total of earned license fees

• Total of external funding from industry

• List of spin-offs

• Involvement in standardizing bodies

• List of positions relevant for transfer

• List of activities in knowledge transfer

14

Assessment in FinlandAija Kaitera

• WHO does it? Academy of Finland / Ministry of Education

• WHEN? Every three years• WHERE? from individual commitees, the Universities• WHY? The state and quality of scientific research in

Finland Funding Allocation

• WHAT? publications, international mobility, develop research training research careers, interaction with society

• HOW? bibliometrics, targeted questionnaires; evaluation of disciplines and research fields (more continuous)

15

Assessment in FinlandAija Kaitera

16

Assessment Activities in FinlandAija Kaitera

• National level research system evaluations by Academy of Finland

• The state and quality of scientific research in Finland (every 3 years; bibliometrics, targeted questionnaires)

• Evaluation of disciplines and research fields (more continuous)

• Organisation Level by Scientific Advisory Boards • Intl evaluation of organisation’s activities (every 3 or 6

years)• Operational level by organisation’s internal

committees• Allocation of basic funding (from ministry to

universities every three years; universities to faculties every year)

• Allocation of quality funding• Evaluation of grant applications (continuous)• Evaluation of research programmes (continuous)

17

Assessment Activities in FinlandAija KaiteraData Collecting Processes

• From researcher to funding bodies• From researcher to university• From university to funding body• From university to ministry

18

Assessment Activities in FinlandAija KaiteraMethods• New national data warehouse (2013?)

• Research: Project, Grant decision, Person, Person’s education, University or Polytechnic, Address, Research group, Space, Publications, Innovations, Other research activities, Results

• Also person, employment, academic data• New national publication database (2012?)

• New national publication classification• Open access is included in national reporting (is/is

not), separate service for full texts • Will include publication quality assessment

“Norwegian model

National annual reportingQuestionnairesGrant reportsSABs

19

Assessment Activities in FinlandAija KaiteraPublication classification in Finland 2010->

20

CERIF4REF schema – 3:Tables generated in CERIF

cfOrgUnit-CORE

cfPers-CORE

cfFundProg-2ND

cfPersName-ADD

cfOrgUnitId_OrgUnit-LINK

cfPersName-OrgUnit-LINK

cfOrgUnit_ResPubl-LINK

cfPers_ResPubl-LINK

cfResPubl_Class-LINK

cfOrgUnit_FundProg-LINK

cfResPublBiblNote-LANG

cfResPublAbstr-LANG

cfClassTerm-LANG

cfOrgUnitResAct-LANG

cfOrgUnitName-LANG

cfPers_Pers-LINK

cfPers_ExpSkills-LINK

cfExpSkillsDescr-LANG

cfResPubl-RES

21

Expert Group on Assessment of University-Based Research

EU, 2010

http://ec.europa.eu/research/science-society/document_library/pdf_06/assessing-europe-university-based-research_en.pdf

22

Purpose Research productivity Quality and scholarly impact

Innovation and social benefit

Sustainability and scale

Research infrastructure

Allocate resources Research output/ bibliometric data

Citation dataPeer reviewKeynote, awards, etc.

Research income ‘research active’ as percentage of total academic staff; libraries, equipment etc.

Drive research mission Differentiation

Research output/ bibliometric dataOutput per research academic

Peer reviewSelf-evaluation

Ration of research income: teaching incomeExternal research income

Ratio of undergraduate: master/PhD students

Increase regional/ community engagement

Publications, Policy, Reports etc.

End user reviewsKeynote, Media awards etc.

Percentage funding from end-usersPatents, licences, spin-offs

Number of collaborations and partnerships

Improve research performance

Research output/ bibliometrics data

Citation dataNumber and percentage publication in top-ranked, high impact journalsPeer review

Assess value-for-money or cost-benefit of research

Research output/ bibliometrics dataOutput per research academic

Peer review and/or citation dataCommercialisation data

Social, economic, cultural and environmental impacts/ benefits indicatorsExternal research incomeEmployability of PhD graduates

Number of collaboration and partnerships

Encourage international co-operation

Research output/ bibliometrics data with focus on European & international collaborations

Percentage of research income from international sources

Number of collaboration and partnerships

Increase multi-disciplinary research

Research output/ bibliometrics data with focus on interdisciplinary fields

Peer reviewSelf-evaluation

New research fields, interdisciplinary teaching programmes, etc.

Research conducted by people from different disciplines

Multidimensional research assessment matrix

23

Purpose of research assessment • Allocate resources

• Drive research mission differentiation

• Increase regional/ community engagement

•Improve research performance

•Assess VFM or cost-benefits of research

•Encourage international co-operation

•Increase multidisciplinary research

24

Next steps

• Report back to this conference

• Report to CERIF Task Group

• etc

25

Thank you

Stephen Grace

Centre for e-Research

King’s College London

http://www.kcl.ac.uk/iss/cerch/

stephen.grace@kcl.ac.uk

Workshop Results

- Many countries deal with the same problems

- Different states of progress to implementation

- Currently, most go for Output Measurement rather than Impact (because measurable)

26

Workshop Results

- CERIF: Quite powerful representation means with respect to Assessment Requirements (-> REF)

- Best Way utilize CERIF is Best Practice

- Qualitative / Quantitative Data Requirements (across European activities)

27

Workshop Results

- Formalization and Measurement is necessary in order to do Analysis

- also the Measurement of soft qualities has to be formalized for Measurement

- CRISs the place to store even soft facts

28

Workshop Results

- CRISs the place to - document / archive soft facts (impacts) - for Reporting / Strategy / Planning …

29