Evaluation of Project Proposals

Junior Project Management course

13-15 April 2011

Budapest, Hungary

Ms. Gabriella Lovasz

Assistant Managing Director

Europa Media

� Who are the Evaluators?

� Balance between

• Academic and industrial expertise

• Geographical representation of Europe

• Gender

� Expertise in the following fields

• Research in the given scientific area

• Industrial/commercial applications

• International project design and implementation

• Administrative, policy-oriented or other

knowledge as required by the Call

Selection of the Experts – Who?

Register as an ”expert”:

https://cordis.europa.eu/emmfp7/

Proposal

EligibilityPre-check by

Commission

Individual

evaluation

Individual

evaluation

Individual

evaluation

Consensus meeting

Scores, ESR

Tresholds

Panel reviewwith

Hearing (optional – IP/NoE)

Commission

Ranking

Applicants informed of

results

ESR

Ethical

review if

needed

The Evaluation Process

� Administrative check by the EC

� Evaluation by the experts:

� Briefing for the experts

� Individual evaluation

� Consensus evaluation

� Final ranking

� Contract negotiations start

On-site vs Remote Evaluation

Background check can be

done – better

understanding

Controlled confidentiality

Better focus

Benefit

Only for the consensus and

panel meeting

FullyStay in Brussels

UnlimitedNo accessAccess to extra resources

(Internet)

Open (but maximised by

payment)

LimitedTime available for

individual evaluation

450 EUR/day450 EUR/dayPayment

Applies

No control

Applies

Controlled

Confidentiality

RemoteOn-site

Briefing for the Evaluators

� Introduction to the Work Programme objectives

� Introduction to the Guide for Applicants and formal criteria

� Overview on the general evaluation principles, and the details of the evaluation

process

� Overview on the scores and the definitions of „Excellent” and „Poor”, etc.

� Introduction to any special EC priorities/issues

� Clarify the possible “conflict of interest”

� Discussions

� Evaluation principles:

� Excellence

� Transparency

� Confidentiality

� Fairness & Impartiality

� Ethical and Security considerations

� Efficiency and Speed

� The evaluators work independently

• No communication is allowed

• Computer is provided if needed (personal laptops are not allowed)

• The internet might be used (depending on the programme)

� Each proposal is evaluated by 3-5 evaluators

� Each evaluator fills out the „Individual Assessment Report” (IAR) forms

Individual Evaluation

Individual Evaluation

� 2-4 proposals per day

� Max 2 hours per proposal

� First impressions

� Title

� Summary

� Objectives

� Partnership

� Consistency & formatting

� Length

� First 15-30 minutes of the evaluation are crucial

Evaluation Criteria

� „Within scope” assessment

� First Reading = First impression

� = ABSTRACT, OBJECTIVES,

Partner list

� Detailed reading

� Read specific chapter for the

specific criterion (find it)

� Make rough notes

� Decide score

� Complete IAR

� Technical issues

�Management

� Finances/Resources

� Impact

Evaluation Criteria - CP

• Contribution, at the European

[and/or international] level, to the

expected impacts listed in the work

programme under the relevant topic/activity

• Appropriateness of measures for

dissemination and/or exploitation of

project results, and management of

intellectual property.

• Appropriateness of the

management structure and

procedures

• Quality and relevant experience of the individual participants

• Quality of the consortium as a

whole (including

complementarities, balance)

• Appropriateness of the allocation and justification of the resources to

be committed (staff, equipment, …)

• Soundness of concept, and quality

of objectives

• Progress beyond the state-of-the-art

• Quality and effectiveness of the support action mechanisms, and

associated work plan

• Appropriate comparative perspective

in relation to the proposed research

IMPACT“Potential impact through the

development, dissemination and use of project results”

IMPLEMENTATION“Quality and efficiency of the

implementation and the management”

S/T QUALITY“Scientific and/or technological

excellence(relevant to the topics addressed by

the call)”

From the Guide for Applicants:

Evaluation criteria applicable to

Coordination and support actions (Supporting)

Individual Evaluation Report

In FP7: a 4-5 page document completed by each expert

� I. Evaluation summary

� II. Recommendation

� Criteria:

� S/T QUALITY ���� “Scientific and/or technological excellence

(relevant to the topics addressed by the call)”

� IMPLEMENTATION ���� “Quality and efficiency of the implementation and the management”

� IMPACT ���� “Potential impact through development, dissemination and use of project results”

1510Total

53Impact

53Implementation

53Scientific quality

MaxThresholdCriterion

Marking System

� 0-5 points awarded for each main

and sub-criteria (0-Fail/Poor, 5-

Excellent)

� Half points can be given!

� Detailed explanation required

• should be given by the evaluators, not only simple scores

� Thresholds for each main criterion

� Threshold for the overall mark

� This can be more than the sum of

the thresholds of the sub-criteria!

Excellent. Any shortcomings are minor

Very good. Certain improvements are still possible

Good. Addresses well, but improvements are

necessary

Fair. Broadly addresses the criterion, but ….

Poor. There are serious weaknesses

Fails to address the criterion or cannot bejudged

5

4 – 4,5

3 – 3,5

2 – 2,5

1 – 1,5

0

Scores

Individual Evaluation - Results

� Total score, overall comments

• for the whole project (strengths and weaknesses, overall recommendations)

� Qualification

• recommended or not recommended for funding – each threshold must be reached!

� The amount of the requested funding is also evaluated: realistic or too much, should be reduced, adequate to the work plan, etc.

� When all experts have finished, the proposal is ready for Consensus evaluation

� Experts meet together to come to a consensus

� Minutes of the meeting are recorded by the „Rapporteur”

� The Commission representative is the „Moderator”

� Experts have a few minutes to read and understand each others comments

� Preliminary discussions followed by detailed assessment of all criteria

The Consensus Meeting

The Consensus Meeting

� Roundtable discussion on each criteria:• The higher/lower marks of an expert have to be explained and justified

• Discussion. Arguments - counterarguments

� Consensus must be obtained for • Scores for each main criterion (the consensus score is NOT the mathematical

average of the individual evaluators score)

• Text of explanation and justification

� The scores of all criteria + the corresponding text agreed upon: IT IS A CONSENSUS

� Consensus Report – drafted by the “Rapporteur”

If There Is No Consensus:

1. New evaluator(s) can be appointed:

Individual evaluation

2. Extended Consensus Meeting with the involvement of the new evaluators

3. If there is no consensus afterwards:

Decision by majority of votes

Goals of the Panel Meeting

� To assess and compare the consensus reports of the different sub-panels:� Overall quality under the different topics

� Number of proposals evaluated

� Special attention to� Proposals that scored very high but failed on one criteria

� Proposals with equal score near the funding thresholds

� Final ranking according to scores:� Established for each topic

� Established for the programme as a whole

� Proposals with equal scores will be ranked:� Objectives, Relevance, Impact

� Horizontal issues

Final Decision, Negotiation

� Commission prepares the following lists:

• Ranking list of the eligible proposals (over the thresholds)

• List of those proposals for which the negotiation process can start – taking

into account the available total fund for the given panel

• Reserve list

• in case of withdrawal – if the negotiation is not successful from any

side!

• List of rejected proposals

Thank you for your attention!

info@europamedia.org

© Europa MediaIt is not allowed to use or distribute the content and design

of the presentation without prior agreement.