hand-out for remote evaluators - european commission · 2018. 6. 27. · hand-out for remote...
TRANSCRIPT
1
H2020 FETOPEN-01-2018-2019-2020 (RIA)
Hand-out for Remote Evaluators Remote Phase Cut-off 16 May 2018
Brussels, 22nd June 2018
2
CONTENT
1. FET Open RIA evaluation process ………………………………………………………………………………….….3
2. WHO - Your Role as Remote Evaluator (RE) ……………………………………………………………….……..3
3. WHEN - Tentative schedule ………………………………………………………………………………………..…….4
4. WHAT - your main tasks as Remote Evaluator (RE)..…………………………………………..………….…5
4.1. Conflicts of Interest …………………………………………………………………………………………………..5
4.2. How to write the individual evaluation report (IER)………………………………………………....6
4.3. Implementation of QCs requested modifications ……………………………………………………..9
4.4. Final verification and approval of the ESR ……………………………………………………………….10
4.5. FET evaluation criteria …………………………………………………………………………………………….11
5. Contact details ……………………………………………………………………………………………………………...12
6. APPENDIX …………………..…………………………………………………………………………………….………...13
Where to find documents that you need for your work ……………………………………..13
Frequent Asked Questions ………………………………………………………………………………….13
3
1. FET Open RIA evaluation process
2. WHO - your role as Remote Evaluator (RE)
FET OPEN RIA evaluation process starts with a remote evaluation of each proposal. This remote
evaluation is done by 4 external experts (remote evaluators) that submit their Individual Evaluation
Report (IER). Remote evaluators do their assessment in relation to the evaluation criteria and give
the corresponding scores.
As an independent expert, you evaluate proposals submitted to the call
You are responsible for carrying out the evaluation of the proposals yourself - you are not
allowed to delegate the work to another person!
You must submit your Individual Evaluation Reports (IERs) in the electronic system within the
given deadline
Significant funding decisions will be made on the basis of your assessment
A high quality of the comments of the remote evaluators is essential because these comments will be
included as such in the Evaluation Summary Report (ESR) that is sent back to the applicants. This
means that appropriate actions should be taken in case of poor quality of the IERs.
Confidentiality :
o You must not discuss evaluation matters, such as the content of proposals, the
evaluation results or the opinions of the remote evaluators, with anyone, including other
experts or Commission/Agencies staff or any other person (e.g. colleagues, students…)
not directly involved in the evaluation of the proposal.
o Do not use/disclose personal data (including email addresses) for purposes other than
evaluation's issues.
4
o You must not contact partners in the consortium, sub-contractors or any third parties.
o You must maintain the confidentiality of documents, paper or electronic, at all times
and wherever you do your evaluation work (on-site or remotely).
Delete all confidential documents upon completing your work.
Please do not ask anyone else to help you in the proposal evaluation
o Social media: Take great care not to post pictures or comments on evaluation matters
through social media.
3. WHEN – tentative schedule
Milestones Deadlines
Call cut-off: 16 May, 2018
Remote experts online briefing: 22 June, 2018
Remote evaluation starts: 22 June, 2018
At least 2 (or all) IERs submitted 02 July, 2018
At least 4 (or all) IERs submitted 09 July, 2018
All IERs submitted: 16 July, 2018
All quality recommendations implemented: 27 July, 2018
Remote cross-readings launched 27 July, 2018
Remote cross-readings completed: 23 August, 2018
Panel review in Brussels: 3-7 September, 2018
Result letters sent to applicants by: 15 October, 2018
Grant Agreements signed by: 15 January, 2019
IMPORTANT: PLEASE RESPECT ALL THE DEADLINES YOU WILL BE GIVEN
(although sometimes different from those stated in the automatic messages
from SEP or in SEP)!
5
4. WHAT should be done? Your main tasks as Remote Evaluator (RE)
4.1. CONFLICT OF INTEREST:
Before starting please check your potential Conflict of Interest! The possible situations constituting a
Conflict of interest are explained also in the contract you have signed.
Check your potential Conflict of Interest (COI). Conditions are spelled out in your contract and in the Code of Conduct (Annex 1)):
You may have a Conflict of Interest if you (non-exhaustive list):
were involved in the preparation of the proposal – including as proposal writer or
consultant;
stand to benefit directly/indirectly if the proposal is successful;
have a close family/personal relationship with someone involved in the proposal;
are a director/trustee/partner of an applicant or involved in the management of an
applicant's organisation;
are employed or contracted by an applicant;
are a member of a Horizon 2020 Advisory Group;
are a member of a Horizon 2020 Programme Committee;
are involved in a National Contact Point or are directly working for the Enterprise Europe
Network;
were employed or contracted by one of the applicants in the last 3 years
were involved in research collaboration with any of the scientists in the last 3 years;
were involved in a grant agreement/decision, the membership of management
structures or a research collaboration with an applicant in the last 3 years;
are in any other situation that casts doubt on your impartiality or that could reasonably
appear to do so.
In case you identified a conflict of interest:
You must inform REA as soon as you become aware of a COI before the signature of the
contract, upon receipt of proposals, or during the course of your work
REA will determine if there is a COI on a case-by-case basis and decide the course of
action to follow.
If you knowingly hide a COI, you will be excluded from the evaluation and your work declared null
and void:
a. The allowance/expenses you claimed may be reduced, rejected or recovered
b. Your contract may be terminated
6
4.2. HOW TO WRITE THE INDIVIDUAL EVALUATION REPORT (IER)
For the proposals, for which you do not have a Conflict of Interest: 1. Read the proposal and evaluate it against the evaluation criteria
2. Check to what degree the proposal is relevant to the Call scope (adherence to all FET
gatekeepers): in-scope or out-of-scope
3. Complete an Individual Evaluation Report (IER)
4. Sign and submit the form in the electronic system
5. Implement quality recommendations suggested by the quality controllers
Please find hereafter the scores to be used and their meaning:
0 — The proposal fails to address the criterion or cannot be assessed due to missing or
incomplete information (unless the result of an ‘obvious clerical error’).
1 — Poor: the criterion is inadequately addressed or there are serious inherent weaknesses.
2 — Fair: the proposal broadly addresses the criterion but there are significant weaknesses.
3 — Good: the proposal addresses the criterion well but with a number of shortcomings.
4 — Very good: the proposal addresses the criterion very well but with a small number of shortcomings.
5 — Excellent: the proposal successfully addresses all relevant aspects of the criterion; any
shortcomings are minor.
• the proposal is evaluated as submitted - not on its potential if certain changes
were to be made
• Re-submissions are evaluated as it would be the first time they are submitted,
i.e. avoid comparison with previous versions of the proposal
• If the proposal 1-3 sections exceeds 15 pages, please disregard excess of pages
(the content of these pages should be invisible)
• You should not give 5 if you identify weaknesses
• If no shortcomings are mentioned under a specific criterion, then giving e.g. 3.5
or even 4 would not be consistent at all with your evaluation,
• If you identified some weaknesses preventing the project from achieving its
objectives, then the score (for a given criterion) should stay below threshold
Important
7
When writing each IER, please make sure that you:
Assess only the proposal's explicit content and merits, do not compare with other proposals and
do not infer or deduct what is not written in the proposal
Cover all criteria and address all the aspects of the evaluation sub-criteria
Provide comments on substance; comments should be neither too short nor too long
Do not summarize the proposal in the IER
Be specific in the comments: avoid too general comments giving impression you haven't read
the proposal; comments should be explicit, factual and concrete
The comments should be objective and respectful
Use proper and rich English language, make the comments understandable to the applicants
Assess the weaknesses, shortcomings and strengths of the proposal for each sub-criterion
Do not repeat same shortcoming or weakness in different criterion and sub-criterion
Do not cross reference (e.g. page number) to the proposal
Do not cross reference in the evaluation report (e.g. "Notwithstanding the lack of
interdisciplinarity described above, the research methods are appropriate")
Do not make any recommendations to the consortium to improve the proposal
Avoid naming specific researchers but refer to the institution instead
Avoid subjective statements (e.g. "in my opinion")
Avoid colloquial expressions (e.g. "huge", "hot topic", "There is not much doubt")
Do not express uncertainty (e.g. "It is hard to evaluate if this is justified" , "It seems...", "It
looks like ..."); use "the proposal will" instead of "the project would"; do not formulate questions
in the assessment
Avoid citations
Avoid too rigid comments, categorical statements (e.g. "Info about XYZ is absent").
ETHICS ASSESSMENT is not part of the evaluation!! The evaluation should not include an ethics
assessment. For the proposals selected for funding there will be a special procedure for that.
Do not evaluate proposals more favourably because they participate in the Open Research Data
Pilot, do not penalise them for opting out of the Pilot.
If the proposal is a resubmission, you should not include references to the previous submission.
The proposal should be evaluated as if it was the first time it is submitted.
Do not make any comments under any sub-criterion on:
1. Out-of-scope,
2. operational capacity,
3. Exceptional funding of applicants from non-EU countries & international
organisations
4. Human embryonic stem cells (hESC).
Please, follow the instructions for each case:
1. If you consider the proposal as being out-of-scope
o Check if the content of a proposal corresponds, wholly or in part, to the description of the call. The "Out-of-scope" (eligibility criterion) should be ticked for the proposals outside the scope of the call i.e. “the proposals that, according to the evaluator’s
8
assessments, does not convincingly satisfy all FET gatekeepers as described under this topic”
o If you consider that the proposal is out-of-scope, substantiate this in the dedicated section.
o Even if you consider the proposal "Out-of-scope", the IER still must be completed o If you consider the proposal as being out-of-scope, it should be reflected in the
scores for Criterion 1 (e.g Blue skies research).
2. If you consider that one or more members of the consortium do not have the necessary
operational capacity:
o You should tick "NO" (under corresponding section) only if you conclude, after checking information provided in sections 1, 2, 3 and 4, that the applicant/s do not have the necessary basic operational capacity to carry out their proposed activities and/or they cannot have access to the facilities they describe (you should check the information provided in curriculum vitae or description of the profile of the applicant, relevant publications or achievements; relevant previous projects or activities, description of any significant infrastructure or any major items of technical equipment).
o Otherwise tick "YES"
3. If the proposal requests exceptional funding for partners from:
a. Third countries not listed in Section A of the General Annexes to the Work
Programme
b. International organizations the majority of whose members are not Member states
or associated countries, and whose principal objective is not to promote scientific
and technological cooperation in Europe
please, assess whether the funding of this entity/s is essential for the action and include
a detailed justification in the dedicated section.
Their participation is considered essential for carrying out the action on the grounds that
participation by the applicant has clear benefits for the consortium, such as:
outstanding competence/expertise
access to research infrastructure
access to particular geographical environments
access to data.
4. Use of human embryonic stem cells (hESC).
o Please, assess whether the proposal intends to use hESC and if applicable, comment on the hESC under the dedicated section
Remember not to introduce comments regarding any of the above 4 points under any sub-
criterion
Please contact REA staff if you find any factual error in these sections or if you have any
doubt on these 4 cases.
9
Assign a score to each criterion:
The score must objectively reflect the aggregated comments under the respective criterion (all
sub-criteria considered together).
Employ the whole range of scores; half-marks can be used (except 0.5); the score 0 should be
used in clearly exceptional cases; there is no score 0.5;
Proposals should not be rewarded / penalised for the same issue twice.
The score should not be influenced by Ethics, participation of third countries not listed in
Section A of the General Annexes, or by (non-)participation in the Open Research Data initiative!
Submit your IER:
Respect the deadlines given by the PO for the remote evaluation.
Once submitted, the IER becomes visible for Quality Control (QC).
Check your email everyday.
4.3. IMPLEMENTATION OF QCs REQUESTED MODIFICATIONS:
In case the QC suggests improvements, the IER will be reopened and you will receive an email
notification.
The QC comments can be seen in SEP, on the left side of the IER's screen (please, press on
'Expand comments') - Figure 1.
Implement the requested modifications within maximum 2 days.
The QC process may be repeated several times until the quality of the quality of IER is
considered appropriate.
Figure 1: The comments in SEP for the quality check: while the repository is keeping all the
comments from the beginning of the work, the left side window displays only the latest 4
comments.
10
Figure 2: The comments in SEP for the quality check: in expanded mode the left-side window
displays all the comments from the beginning of the work, structured on evaluation criteria.
4.4. FINAL VERIFICATION AND APPROVAL OF THE ESR
• Once all four IERs have been drafted, quality checked, and submitted, you will receive access
to the consolidated report built from the four IERs (the Evaluation Summary Report or ESR). In the
ESR all the comments are collated per sub-criterion in an anonymous way.
• At this time you will receive an invitation from the responsible PO to view the ESR. In order
to do this, please go to SEP
o From the top row menu select "Proposals" then in the "Call" filter enter the indicative of the
current call H2020-FETOPEN-2018-2020-RIA_16-01-2018. You may search the proposals either by
acronym or number, using the filters "Acronym" or respectively "Proposal".
o Click on the "R" icon under the column ESR
o Click on the "View Tasks" button in the bottom side of the window. This will bring the screen
with the ESR.
11
o In the ESR window press the "Expand" button. This will allow you to see the comments
provided by all four remote evaluators
• At this stage you will have the opportunity to check the complete set of comments for the
proposal under consideration and to confirm your comments and scores.
• In really exceptional cases, when you discover that you have overlooked a crucial specific
point or you have made a substantial mistake, please inform the POs responsible for the cluster, the
proposal under consideration belongs to. This will mean that you are requesting reopening of the IER
and an opportunity to implement corrections in your own comments and/or scores.
• Do not comment in your revision on other evaluators' IERs or on your own opinion from
before the revision! Your evaluation task is still an individual and independent one.
• Please do not ask for reopening for minor points or for errors you believe have been made by
the other evaluators!
NOTE: Should this operation be required in case of one or several of your IERs, you will receive
detailed instructions from the Call Coordinator and the POs.
4.5. FET EVALUATION CRITERIA
Excellence Impact Quality and efficiency of the implementation
Adherence to the "FET gatekeepers" as described in the call text:
Clarity of the radical vision of a science - enabled technology and its differentiation from current paradigms.
Novelty and ambition of the proposed science-to-technology breakthrough that addresses this vision.
Range of and added value from interdisciplinarity for opening up new areas of research; non - incrementality of the research proposed.
High-risk, plausibility and flexibility of the research approach.
The extent to which the outputs of the project would contribute to the expected impacts listed in the work programme under this topic.
Effectiveness of measures and plans to disseminate and use the results (including management of IPR) and to communicate about the project to different target audiences.
The following aspects are taken into account:
Coherence and effectiveness of the research methodology and work plan to achieve project objectives and impacts, including adequate allocation of resources to tasks and partners
Role and complementarity of the participants and extent to which the consortium as a whole brings together the necessary expertise.
Threshold: 4/5 Weight: 60%
Threshold: 3.5/5 Weight: 20%
Threshold: 3/5 Weight: 20%
12
FET Gatekeepers (specific to the Work Programme 2018-2020)
Radical vision: the project must address a clear and radical vision, enabled by a new technology concept that challenges current paradigms. In particular, research to advance on the roadmap of a well-established technological paradigm, even if high-risk, will not be funded.
Breakthrough technological target: the project must target a novel and ambitious science-to-technology breakthrough as a first proof of concept for its vision. In particular, blue-sky exploratory research without a clear technological objective will not be funded.
Ambitious interdisciplinary research for achieving the technological breakthrough and that opens up new areas of investigation. In particular, projects with only low-risk incremental research, even if interdisciplinary, will not be funded.
5. Contact details
Barbara GERRATANA [email protected]
A Patrizia Tenerelli [email protected]
Marco Giorgini [email protected]
Maksym Tsytlonok [email protected]
B Linda Curzola [email protected]
Christiane Wilzeck [email protected]
Antonio Loredan [email protected]
C Arianna DELLACA [email protected]
Antonios Fysekidis [email protected]
Adelina Nicolaie [email protected]
D Olivier Dahon [email protected]
Adriana Godeanu Metz [email protected]
Salvatore Spinello [email protected]
E Iliana Kostova [email protected]
Antonios Fysekidis [email protected]
13
6. APPENDIX
Where to find documents that you need for your work
• Work Programme 2018-2020:
http://ec.europa.eu/research/participants/data/ref/h2020/wp/2018-
2020/main/h2020-wp1820-fet_en.pdf , including the General Annexes:
https://ec.europa.eu/research/participants/data/ref/h2020/other/wp/2018-
2020/annexes/h2020-wp1820-annex-ga_en.pdf
• Admissibility and Eligibility conditions:
https://ec.europa.eu/research/participants/data/ref/h2020/other/wp/2018-
2020/annexes/h2020-wp1820-annex-ga_en.pdf (sections B and C)
• The list of Countries directly eligible for funding:
https://ec.europa.eu/research/participants/data/ref/h2020/other/wp/2018-
2020/annexes/h2020-wp1820-annex-a-countries-rules_en.pdf
• Proposal template:
http://ec.europa.eu/research/participants/data/ref/h2020/other/call_ptef/pt/h2
020-call-pt-fetopen-ria-2018-20_en.pdf
• "Evaluation Service - IT User Manual Individual Evaluation Report":
https://ec.europa.eu/research/participants/data/support/expert/expert_evaluati
on_user_manual.pdf
• Frequently Asked Questions
Please visit the FAQ section of the Participant Portal (Search for the call topic FET-
Open: http://ec.europa.eu/research/participants/portal/desktop/en/support/faq.html