european research council · 2017. 3. 10. · european research council established by the european...

The European Research Council

ERC Monitoring & Evaluation

Strategy

European Research Council

Established by the

European Commission

Outline of the presentation

Strategy for Monitoring and Evaluation of ERC funding

activities

Implementing the strategy

CSA projects

│ 2


Established by the

European Commission

Increasing pressure to account for public

funds

Accountability in Principal/Agent Model ─ Principal : right to monitor (ensure that agent acts in the interest of

Principal)

─ Agent : obligation to account for resources

Evolving “accountability” ─ Core question : “what have we done with the money”

─ Traditionally :

─ “Which projects do you fund?”

─ Currently : increased expectations

─ “which results have you achieved?”

─ “what is their significance- for science, economy, society” ?

─ Changing question

─ not if R&D investments produce benefits

─ but how are your investments produce benefits (or contribute to

producing benefits) │ 3


Established by the

European Commission

│ 4

ERC Monitoring and Evaluation (M&E) Strategy

• ERC implements the "Ideas" Specific Programme of the EU's 7th

Research Framework Programme. FP evaluated by EC.

• Roadmap of the Evaluation of the 7th FP by the EC

Progress Report - 2009

Interim Evaluation - 2010

Ex-post Evaluation – 2015

• Coordination: EC - Unit A.5 Evaluation


Established by the

European Commission

│ 5

Role of ERC M&E

Monitoring

Inform decision making of the ERC council

and ERCEA senior management

Evaluation

Provide evidence base for the ERC Assessment

(mid-term review and the ex-post evaluation)

Inform stakeholders, partners and the

public on ERC activities, performance and impact

(incl. reporting obligations of the ERC/ERCEA)


Established by the

European Commission

ERC Approach – I

│ 6

Direct impacts

Accountability

& Information

Science

Management Management

Efficiency

Advancing

knowledge &

dissemination

Impact on

researchers

Impact on

research

organisations

Emerging

research

areas

Impact on

research

policies and

funding

structures

Derived impacts

Structural impacts

Training

Economic

benefits

Societal

benefits

Performance

Attract best researchers

and industrial research

investment

Reinforce excellence,

dynamism and creativity

Stable, effective

administration

Better exploit research

assets and foster

innovation towards a

dynamic knowledge

based society

Dimensions

Components

Objectives


Established by the

European Commission

ERC Approach - II

│ 7

It takes time for results to be produced and years for the impact to unfold.

Only rigorous and carefully designed studies can help uncover the impact

External Expertise

Internal capacities

Call for proposals

Call for tenders

Expert Groups

Independent academics

(to be explored)

Good research information play a critical role


Established by the

European Commission

│ 8

Implementation of ERC M&E Strategy

Challenges

• Managing the limited resources the ERC can devote to M&E activities

• Securing the high quality data required to deliver the strategy

• “over-fishing“ and “evaluation fatigue”

• Data quality

• Adequate methodologies to measure impacts

• impact not easy to measure and/or to attribute

• impact of ERC funding will certainly need some time to

emerge into recognizable effects


Established by the

European Commission

│ 9

Data sources

• Process-produced data (in-house data)

• External data:

Collected by CSA funded projects

Collected from specialized databases (e.g. bibliographic systems such as

Web of Science, Scopus …)

Data collected from other sources (Eurostats, Erawatch ...)


Established by the

European Commission

│ 10

Data Analysis Tools

• AVEDAS – ERC research information system:

Integration of different (internal & external) data sources through user

analytics module

Enabling basic analysis in support of monitoring and evaluation of ERC

operations

Flexible and open for new data and analysis developments, for example

integration of bibliometric tools developed by CSA projects


Established by the

European Commission

│ 11

• Coordination: Professor Stefan Hornbostel (IFQ)

• Aim:

to understand the impact of the grants in fours areas:

(1) career development

(2) institutional environments (e.g. on factors which favour or hinder

the establishment of independent research groups);

(3) structural effects in the European landscape (mobility) and

(4) contribution to academic advancement

(i.e. impact of publications of grant holders on the advancement of

their fields).

Monitoring European Research Council‘s

Implementation of Excellence [MERCI]


Established by the

European Commission

│ 12

• Coordination: Dr Maria Nedeva, Manchester Institute of Innovation Research (MIoIR), University of Manchester, UK

• Aim: develop and apply a methodology to measure and attribute impacts of the two ERC funding schemes in three areas:

(1) on researchers (career path, skills, networking possibilities)

(2) on research organisations and

(3) on national as well as European research funding organisations.

Understanding and Assessing the impact and

outcomes of the ERC funding schemes

[EURECIA]


Established by the

European Commission

│ 13

• Coordination: Katy Whitelegg, Austrian Institute of Technology

• Aim: compare the ranking of applicants made by the panels (peer-review) and ranking based on the frontier research indicators measured through bibliometrics (publication records and citations from proposal submission)

Development and Verification of a Bibliometric

model for the Identification of Frontier Research

[DBF]


Established by the

European Commission

│ 14

• Coordination: Dr Thomas Reiss, Fraunhofer Institute Systems and Innovation Research, Karlsruhe, Germany

• Aim: to identify topically ‘emerging researach areas’ and assess to which extend the ERC-supported projects cover and to contribute those emerging scientific areas

* Identification of ‘emerging areas’ on basis of bibliometric analysis (SCI, SSCI, A&HICI, years : 1998-2007)

Emerging Research Areas and their Coverage by

the ERC-supported projects [ERACEP]

european research council · 2017. 3. 10. · european research council established by the european...

Documents