evaluation of project proposals in fp7
Post on 05-Dec-2014
3.315 Views
Preview:
DESCRIPTION
TRANSCRIPT
Evaluation of Project Proposals
Junior Project Management course
13-15 April 2011
Budapest, Hungary
Ms. Gabriella Lovasz
Assistant Managing Director
Europa Media
� Who are the Evaluators?
� Balance between
• Academic and industrial expertise
• Geographical representation of Europe
• Gender
� Expertise in the following fields
• Research in the given scientific area
• Industrial/commercial applications
• International project design and implementation
• Administrative, policy-oriented or other
knowledge as required by the Call
Selection of the Experts – Who?
Register as an ”expert”:
https://cordis.europa.eu/emmfp7/
Proposal
EligibilityPre-check by
Commission
Individual
evaluation
Individual
evaluation
Individual
evaluation
Consensus meeting
Scores, ESR
Tresholds
Panel reviewwith
Hearing (optional – IP/NoE)
Commission
Ranking
Applicants informed of
results
ESR
Ethical
review if
needed
The Evaluation Process
� Administrative check by the EC
� Evaluation by the experts:
� Briefing for the experts
� Individual evaluation
� Consensus evaluation
� Final ranking
� Contract negotiations start
On-site vs Remote Evaluation
Background check can be
done – better
understanding
Controlled confidentiality
Better focus
Benefit
Only for the consensus and
panel meeting
FullyStay in Brussels
UnlimitedNo accessAccess to extra resources
(Internet)
Open (but maximised by
payment)
LimitedTime available for
individual evaluation
450 EUR/day450 EUR/dayPayment
Applies
No control
Applies
Controlled
Confidentiality
RemoteOn-site
Briefing for the Evaluators
� Introduction to the Work Programme objectives
� Introduction to the Guide for Applicants and formal criteria
� Overview on the general evaluation principles, and the details of the evaluation
process
� Overview on the scores and the definitions of „Excellent” and „Poor”, etc.
� Introduction to any special EC priorities/issues
� Clarify the possible “conflict of interest”
� Discussions
� Evaluation principles:
� Excellence
� Transparency
� Confidentiality
� Fairness & Impartiality
� Ethical and Security considerations
� Efficiency and Speed
� The evaluators work independently
• No communication is allowed
• Computer is provided if needed (personal laptops are not allowed)
• The internet might be used (depending on the programme)
� Each proposal is evaluated by 3-5 evaluators
� Each evaluator fills out the „Individual Assessment Report” (IAR) forms
Individual Evaluation
Individual Evaluation
� 2-4 proposals per day
� Max 2 hours per proposal
� First impressions
� Title
� Summary
� Objectives
� Partnership
� Consistency & formatting
� Length
� First 15-30 minutes of the evaluation are crucial
Evaluation Criteria
� „Within scope” assessment
� First Reading = First impression
� = ABSTRACT, OBJECTIVES,
Partner list
� Detailed reading
� Read specific chapter for the
specific criterion (find it)
� Make rough notes
� Decide score
� Complete IAR
� Technical issues
�Management
� Finances/Resources
� Impact
Evaluation Criteria - CP
• Contribution, at the European
[and/or international] level, to the
expected impacts listed in the work
programme under the relevant topic/activity
• Appropriateness of measures for
dissemination and/or exploitation of
project results, and management of
intellectual property.
• Appropriateness of the
management structure and
procedures
• Quality and relevant experience of the individual participants
• Quality of the consortium as a
whole (including
complementarities, balance)
• Appropriateness of the allocation and justification of the resources to
be committed (staff, equipment, …)
• Soundness of concept, and quality
of objectives
• Progress beyond the state-of-the-art
• Quality and effectiveness of the support action mechanisms, and
associated work plan
• Appropriate comparative perspective
in relation to the proposed research
IMPACT“Potential impact through the
development, dissemination and use of project results”
IMPLEMENTATION“Quality and efficiency of the
implementation and the management”
S/T QUALITY“Scientific and/or technological
excellence(relevant to the topics addressed by
the call)”
From the Guide for Applicants:
Evaluation criteria applicable to
Coordination and support actions (Supporting)
Individual Evaluation Report
In FP7: a 4-5 page document completed by each expert
� I. Evaluation summary
� II. Recommendation
� Criteria:
� S/T QUALITY ���� “Scientific and/or technological excellence
(relevant to the topics addressed by the call)”
� IMPLEMENTATION ���� “Quality and efficiency of the implementation and the management”
� IMPACT ���� “Potential impact through development, dissemination and use of project results”
1510Total
53Impact
53Implementation
53Scientific quality
MaxThresholdCriterion
Marking System
� 0-5 points awarded for each main
and sub-criteria (0-Fail/Poor, 5-
Excellent)
� Half points can be given!
� Detailed explanation required
• should be given by the evaluators, not only simple scores
� Thresholds for each main criterion
� Threshold for the overall mark
� This can be more than the sum of
the thresholds of the sub-criteria!
Excellent. Any shortcomings are minor
Very good. Certain improvements are still possible
Good. Addresses well, but improvements are
necessary
Fair. Broadly addresses the criterion, but ….
Poor. There are serious weaknesses
Fails to address the criterion or cannot bejudged
5
4 – 4,5
3 – 3,5
2 – 2,5
1 – 1,5
0
Scores
Individual Evaluation - Results
� Total score, overall comments
• for the whole project (strengths and weaknesses, overall recommendations)
� Qualification
• recommended or not recommended for funding – each threshold must be reached!
� The amount of the requested funding is also evaluated: realistic or too much, should be reduced, adequate to the work plan, etc.
� When all experts have finished, the proposal is ready for Consensus evaluation
� Experts meet together to come to a consensus
� Minutes of the meeting are recorded by the „Rapporteur”
� The Commission representative is the „Moderator”
� Experts have a few minutes to read and understand each others comments
� Preliminary discussions followed by detailed assessment of all criteria
The Consensus Meeting
The Consensus Meeting
� Roundtable discussion on each criteria:• The higher/lower marks of an expert have to be explained and justified
• Discussion. Arguments - counterarguments
� Consensus must be obtained for • Scores for each main criterion (the consensus score is NOT the mathematical
average of the individual evaluators score)
• Text of explanation and justification
� The scores of all criteria + the corresponding text agreed upon: IT IS A CONSENSUS
� Consensus Report – drafted by the “Rapporteur”
If There Is No Consensus:
1. New evaluator(s) can be appointed:
Individual evaluation
2. Extended Consensus Meeting with the involvement of the new evaluators
3. If there is no consensus afterwards:
Decision by majority of votes
Goals of the Panel Meeting
� To assess and compare the consensus reports of the different sub-panels:� Overall quality under the different topics
� Number of proposals evaluated
� Special attention to� Proposals that scored very high but failed on one criteria
� Proposals with equal score near the funding thresholds
� Final ranking according to scores:� Established for each topic
� Established for the programme as a whole
� Proposals with equal scores will be ranked:� Objectives, Relevance, Impact
� Horizontal issues
Final Decision, Negotiation
� Commission prepares the following lists:
• Ranking list of the eligible proposals (over the thresholds)
• List of those proposals for which the negotiation process can start – taking
into account the available total fund for the given panel
• Reserve list
• in case of withdrawal – if the negotiation is not successful from any
side!
• List of rejected proposals
Thank you for your attention!
info@europamedia.org
© Europa MediaIt is not allowed to use or distribute the content and design
of the presentation without prior agreement.
top related