1

May 11, 2015

The evaluation of research units at

HCERESAlain Cappy, coordinator for the S&T domain

2

Outline

1-HCERES présentationStatus and OrganizationFounding PrinciplesMissions and ObjectivesEU RecognitionAnnual WorkIntegratedE

2-Evaluation of research unitsPrinciplesReportSyntheses

3- Conclusion

3

HCERES: Status and Organisation

Status: Independent Administrative Authority

Organization•President, chairs the board •Board of the Council: 30 members•3 evaluation departments

• Institutions• Research units• Programs and degrees

•Observatory of Science and Technologies ( data base, bibliometric analysis, indicators…)•Support service

4

HCERES: Status and Organisation

Scientific organization• About 100 part-time scientific delegates• 4,500 experts commited every year, including 20% outside France

Administrative organization•About 100 full-time administrative and technical staff

Budget in 2015: about 15 M€

5

Founding Principles

Independence: Status of independent administrative authority. HCERES to carry out its missions without coming under pressure from:•government authorities, •Evaluated entities ( research units, Universities, research bodies),•or any other stakeholders

HCERES guarantees compliance with the rules of ethics, in particular those concerning conflicts of interests.

6

Founding Principles

Competence of experts and transparency of proceduresQuality of evaluation rests on:• the expert’s skills and correct matching his/her profile with the entity evaluated,• a methodology and procedures guaranteeing equality of treatment,• drawing up by HCERES of a quality management system.

Transparency of evaluation lies on publication (website) of :• criteria for and methods of evaluation,• the list of scientific delegates, experts and their positions,• Summary of evaluation reports.

7

HCERES: Missions and Objectives

Evaluating: •HE & R institutions (universities, schools and research bodies), taking account of all their missions and activities.•Research activity laboratories and research units of these institutions• Education programs and degrees in the French higher education system (BMD)

Assisting evaluated entities in the continuous improvement of their practices

Providing clarification for decision-makers:•The State (financing, recognition of qualifications)•Research bodies (certification of laboratories)•Others (Local Authorities, Companies, Families)

8

What HCERES Do and Do Not

HCEES evaluates and does NOT decide

Decisions about accreditation and funding belong to Universities, Institutions, State,…

HCERES does NOT evaluate persons

But representatives of person evaluation bodies are included at different levels :

o HCERES Councilo Visiting committees

9

European Recognition

HCERES is member of the European Association for Quality Assurance in Higher Education (ENQA). ENQA promotes European co-operation in the field of quality assurance in higher education .

HCERES is listed in EQAR, the European Quality Assurance Register for Higher Education.

10

Annual Work

Institutions:•60-70 universities and schools•3 research organisms

Research units: •600 evaluation committees

Training courses:•700 Bachelor degrees •300 Master degrees•50 doctoral schools

11

A 5-Year Period

The Regional Breakdown of the Evaluation Campaigns

12

Integrated Evaluation

Evaluation of research units including on site visit

For one institution ( University, Engineer School…)

Evaluation of training programmes, ( Bachelors, Masters, Doctoral Schools)

Evaluation of institution including on site visit.

Evaluation reports

Evaluation reports

13

Document Management System

Web based evaluation process (GED)• File and report deposition• Evaluation committee creation• Flags to follow the state and progress of a

specific unit evaluation

14

Document Management System (GED)

15

Evaluation of research units

16

Methodological Principles

The criteria defined and procedures implemented by the High Council take into account:•the diversity of the natures and missions of the research units•the diversity of disciplinary fields ( Life Sciences, Social Sciences and Humanities, Physical and Engineering Sciences)•based on peer-assessment (national and international experts)

Evaluation goals:•Oriented towards improvement of unit performance•Combines quantitative data ( scientific production, impact) with qualitative information assessed by experts

17

Evaluation of Research Units: for Who ?

the recipients of the evaluation are:• Research units and researchers• research organisms, Universities, Grandes Ecoles• Students• Local Authorities (regions, etc.) • Ministry for Higher Education and Research and other ministries

18

Evaluation of Research Units

The annual task: evaluation of all State funded research units in specific Regions ( France is divided in 5 evaluation campaigns)

• ~ 600 research units• > 3000 experts committed every years for the peer review

Covering ALL research bodies supported by public funds, including private universities, architectural and cultural research, and possibly clinical research…

19

Main steps

1.The research unit writes a report following a model. This report

is deposited on the HCERES document management system

(GED).

2.A HCERES scientific officer is designed to manage the

evaluation

3.An evaluation committee is appointed by HCERES

4.The committee prepares a draft of the report before the visit

5.A visit (1, 2 or 3 days ) of the research unit is organised. During

the visit, the unit presents its main results and strategy. The

committee has meeting with the unit personnel

Research Unit Evaluation Process

20

Main steps (cont’d)

6.After the visit the committee finalised the report

7.The report is controlled by HCERES to be sure that it follows HCERES writing rules

8.The report is transmitted to the unit and its supervising institutions. A summary of the report is published on the HCERES website.

Research Unit Evaluation Process

21

The Evaluation Report

The report is the main output of the evaluation.

• The report should be critical, not descriptive,

• The report must contain evaluation details of the research

unit at a whole but also for each research team,

• The report has to specify strengths, weaknesses and

recommendations.

22

Evaluation Report

• The evaluation report: the six evaluation items

• Scientific outputs and quality of research• Academic reputation and appeal• Interactions with the social, economic and cultural

environment• Organization and life of the unit• Involvement in training• Strategy and five-year plan

•These six items are used both for the research unit as a whole and for the research groups

23

Evaluation of research units

Synthesis by geographical site:In addition to the research units evaluation reports, HCERES writes research syntheses : Goal: To give a schematic view of the particularities of a geographical site based on:•The characterization of working forces in the three domains (Social sciences and Humanities (SH), Life sciences (LS),Physical and Engineering Sciences (PE))•The analysis of evaluation reports in each domainThe synthesis contains two parts

- the workforce- the synthesis of the evaluation reports

Specific syntheses can be produced by order

24

Breakdown of workforce: Grenoble

25

Breakdown of workforce: Grenoble

26

Open questions

Evaluation is a difficult task. Many questions are still open

•How to evaluate interdisciplinary research ?

•How to measure the impact of a research ?

•How to efficiently use bibliometric data

HCERES working groups and seminars are organized to tackle these questions

27

Conclusion

• HCERES, a ‘High Council for Research and Higher Education’ devoted in France to the evaluation of Institutions, research units and training courses

• HCERES is an independent authority• HCERES performs an integrated evaluation in

each site• HCERES is member of ENQA and EQAR• All the French research units and research

groups are evaluated every 5 years