dir s erc/european commission rtd, directorate s

Dir SERC/European Commission

RTD, Directorate S

The European Research Council

The evaluation of the ERC-2007-StG Call

IDEAS Programme Committee, January 31, 2008

European Research Council

│ 2

Overview

1. Status of evaluation report

2. Key data – both stages

3. The eligibility process

4. Budgets by domains and panels

5. The panel meetings and the ranked lists per panel

6. The consolidation by the Panel Chairs

7. Recommendations on grant levels

8. Feedback to applicants

9. Status of redress


│ 3

Status of evaluation report

Will be conform to that of other programmes

• Offering same facilities to PMC members

• We did not succeed having a fully quality checked document for this meeting. However, key data tables are distributed

Formal E.R. statistical reporting will be based on the sample of 201 proposals in the main list – by convention

Informal reporting, including public dissemination, is based on a sample of the top 300 proposals

This is a reasonable estimate of the number that may be funded


│ 4

Key data- number of proposals by evaluation step

9167

8794

559

554

547

368 ineligible

5 withdrawn

8235 rejected

5 not submitted to second stage

2 passed away, 4 ineligible

1 withdrawn

Submitted stage 1

Evaluated stage 1

Selected stage 1

Submitted stage 2

Evaluated stage 2

201 in main list

116 in reserve list

113 reserve: reject for no budget

117 rejected: below thresholds


│ 5

Key data: Evaluation processStage 2: 559 proposals expected

Eligibility and withdrawals

Reception of proposals

INDIVIDUAL ASSESSMENTS

Panel members Referees

INTERVIEWS

20 panel-ranked lists

PANEL MEETINGSPANEL CHAIR

MEETING

Single Consolidated

list

1 day

554 7

547

2 to 3 days


│ 6

The eligibility process

Eligibility decisions have been taken conform the standard FP7 methodology - and in accordance with its ‘case-law’

Some of the decisions have been complex, and have required discussion in parallel with the review

We attempt to minimise the number of such cases

However, given the numbers and the time-constraint, partial parallel processing of review and eligibility is necessary – and is in accordance with the Rules


│ 7

Budgets by domains and panels,

based on adopted / published call budget, 289.5M€

80% 20%

8 PE panels 7 LS panels 5 SH Reserve

Nominal panel budgets, proportional to sum of stage-1 requested grant

E.g. PE 1 2 43

E. g. PE4

Fits in nominal panel budget,Score > 9, no ordering

…

Main list Reject

Candidate for reserve budget,Ranked in priority order, with scoring convention

Reserve ………..

289.5M€Additional budget

contributions add to reserve

Additional budget

Not fundable,Score < 8


│ 8

The stage-2 interviews / panel meetings

Panels operated independently, typically 3 days

Interviews with applicants perceived as extremely useful complement to individual assessments

Panels identified four groups of proposals:1. Main list proposals, inside nominal panel budget

2. Reserve list, serious candidate, priority ordering

3. Reserve list, good proposal, clearly outside budget

4. Rejected proposal, failing threshold

Panels tagged inter-disciplinary, cross-panel / domain proposals

For groups 1 and 2, panels recommended the grant level


│ 9

The panel chair consolidation meeting

Purpose of the meeting: to establish a consolidated ranking of the serious candidate reserve (group 2) proposals

1. With special emphasis on inter-disciplinary proposals

2. Given 20 panels across all scientific fields, not a trivial affair ….

3. Because no absolute excellence standard exists: small score differences are meaningless across panels

To faciliate, a starting point ranking was needed1. Not constraining the freedom of the panel chairs

2. ‘Bureaucratically fair’: purely based on an algorithm

3. Accounting for different panel and proposal sizes

4. Without prejudice to any possible differences of excellence between panels


│ 10

The consolidation method (simplified)

By convention, all panels scored their serious reserve candidates at 8.9, and sub-ranked them: first, second, ….

Accum

ulated grant

Norm

alised accum

ulated grant

Normalise each panel on its nominal budget

Res 1Res 2Res 3Res 4Res 5Res 6Res 7Res 8Res 9…..

Tagged as inter-disciplinary

1.0

Nominal budget

Ordering by normalised accumulated grant


│ 11

The final ranking by the panel chairs

Regarding inter-disciplinary proposals, the panel chairs:

1. Considered that they did not have the resources to re-examine these in detail

2. Confirmed their confidence in the prior work of the panels – ‘mainstreaming’

Regarding the ranked list, the panel chairs unanimously adopted the starting point proposed by the Commission as a fair result

This ranking has not been modified, neither by Commission nor by the Scientific Council


│ 12

The issue of recommended grant levels - I

Panels’ remit included providing recommendations on award levels

Individual panels faced some difficulties in this respect:

1. Proposals not overly detailed, over-estimates by PI or host …

2. Incomplete understanding / interpretation of the Rules of Participation

3. Contradictory information during interviews

Driven by this, and by different ‘needs of the field’, panels arrived at different solutions:

1. No grant reductions; reductions across the board; big reductions on some proposals

2. Some panels have removed the PI salary for PI’s with permanent positions – problematic in view of Rules and of grant mobility

3. Some panels have calculated the grant ‘bottom-up’


│ 13

The issue of recommended grant levels - 2

The ‘reductions issue’ was discussed in the Panel chair meeting

1. In general, panel chairs confirmed the positions taken by their individual panels as fair and reasonable in the context of the field

2. A small number of corrections was introduced during the meeting

3. On the specific issue of the PI’s salary, panel chairs realised the difficulty of the situation and recommended that the Commission applies appropriate corrections

The Commission has applied a correction to all cases where the PI’s salary was explicitly removed

The Scientific Council has strongly endorsed the position of the panel chairs, and requested the Commission to award the grants accordingly, without negotiation


│ 14

Towards granting

The Commission has maintained the possibility for successful applicants to seek redress against the level of awarded grant, using the redress procedure

• In reality, no such request for redress has been received

The Commission has been cautious in its feedback to applicants

1.Pending formal decisions on 44.5M€ of third-country contributions

2.In view of the pending redress

Granting has started on the main list – 201 proposals

1.About 40 ethical reviews are ongoing

2.The expectation is that about 300 grants will be awarded


│ 15

Transparency towards applicants

Four messages to the four groups of applicants:

1. 201 proposals in main list: granting is imminent, started for 65

2. 116 serious reserve candidates; all have score 8.9; probability of a grant varies from 1 to 0 down the ranking

3. 113 good proposals but clearly outside budget; score between 8.0 and 8.8; clear information given

4. 117 proposals fail threshold; score < 8.0; clear information given

Subject to individual disclaimers, list of all 430 proposals above threshold is now published on ERC web-site

1. Most of the 116 serious reserve candidates can make a reasonable assessment of their probability


│ 16

Status of redress

The ‘Redress’ procedure has worked well

For Stage-1, 245 requests for redress received, three main areas:1. Eligibility – none sustained

2. Factual errors by reviewers – 15 sustained

3. Scientific judgement – none sustained

15 stage-1 proposals were re-reviewed by panels

1. One applicant has been invited to submit a stage-2 proposal

Stage-2 redress: has just started1. 27 requests were received


│ 17

Details of stage-1 redress

Scientific judgement of panels 150

The review process 201. Wrong panel 6

2. Conflict of interest 14

Very near to threshold 53

Negative, offensive 251. Due to editing / English 18

2. Discriminatory 7

Eligibility 49

Factual errors by reviewers 26

Rebut reviewers / complain size of proposal 6

NOTE: a single redress request may address multiple categories


│ 18

Main lessons from stage-2

Overall, the process worked well: consolidation effective

Interviews successful, to be maintained for StG

Review the framework in which panels operate1. Respect autonomy, specificities of scientific fields

2. But more need for coherent decision-making

The ‘mainstreaming’ approach to inter-disciplinarity has worked, but reflection needed:

1. It does not give much visibility

2. Difficult to achieve inter-panel coherence

Reviewers comments in feedback to applicants:1. Reveals some flawed judgements – redress

2. Transparency inevitably drives improvement

dir s erc/european commission rtd, directorate s

Documents

panel chairsrecommendations

evaluated stage

main list proposals

selected stage

groups of proposals

reserve budget

panelsthe panel meetings

proposals expectedeligibility