horizon 2020 proposal evaluation · pdf filehorizon 2020 proposal evaluation dr alex berry...
TRANSCRIPT
Horizon 2020 Proposal
EvaluationDr Alex Berry
UKRO European Advisor
13 December 2016
• Horizon 2020 evaluation process general overview– How to become an evaluator
– Evaluation criteria
• ERC evaluation process
• Marie Skłodowska-Curie Actions evaluation process
• Further information
Content
Horizon 2020:
EvaluationProcess overview
Overview of the process
Receipt of proposals
Individualevaluation
Consensusgroup
Panel Review Finalisation
Evaluators
IndividualEvaluation
Reports
(Usually done remotely)
ConsensusReport
(May be done remotely)
Panel report
Evaluation Summary Report
Panel ranked list
Eligibility check
Allocation of proposals to evaluators
Final ranked list
Source: European Commission
Information on evaluation outcome: 5 monthsSignature of grant agreement: 8 monthsfrom final date for submission
• Admissibility check:– General conditions on admissibility set out in General Annex B/specific
conditions in Work Programme. Proposals must be:
• submitted in the electronic submission system before the deadline given in the call
• readable, accessible and printable
– Incomplete proposals may be considered inadmissible
• Eligibility check:– General eligibility criteria set out in General Annexes A and C (e.g.
requisite minimum number of legal entities from different Member States or Associated Countries)
– Proposal must correspond to topic description
Receipt of proposals
• Experts read the proposal and evaluate against the evaluation criteria:– without discussing with others
– as submitted, not on potential if certain changes were to be made
• Excess pages marked with a watermark (experts instructed to disregard).
• Check degree to which proposal is relevant to call/topic
• Experts complete Individual Evaluation Report (IER)– Give view on operational capacity
– Give comments and scores for all evaluation criteria
– Do not recommend substantial modifications
Individual evaluation
• Selection criteria:– Financial capacity: in line with financial regulation and rules for
participation
– Operational capacity: assessed ability to carry out the project effectively
• Three Award criteria: Excellence, Impact, Quality and Efficiency of Implementation
• Each criterion scored out of 5. Threshold for each is 3, overall threshold 10 (but note specific conditions in topic descriptions).
• Innovation actions and SME instrument: impact score weighted 1.5
Evaluation
Evaluation process - Consensus
• Panel will review all the proposals within a call, or part of a call to:– ensure consensus groups have been consistent in their evaluations
– if necessary, propose new set of marks or comments
– resolve cases where a minority view was recorded in consensus report
• Panel report includes ‘Evaluation Summary Report’ (ESR) for each proposal and ‘Panel ranked list’
• If necessary, priority order for proposals with same score– Highest priority to proposals addressing topics not otherwise covered by
more highly-ranked proposals
– Highest excellence score*; then highest impact score*; then size of budget for SMEs; then gender balance in project team
Panel review
*For Innovation actions, this order is reversed
• Projects funded according to ranking within budget
• Outcome communicated to proposal coordinators:– If successful → evaluation information letter
– Not successful → proposal rejection letter
– Reserve list
• Proposal coordinators will receive and Evaluation Summary Report (ESR) showing the results of the evaluation
Finalisation
• Most calls will follow a one-stage procedure
• At first stage, only ‘excellence’ and ‘impact’ are evaluated (selected aspects only). Individual thresholds for each criterion set at 4. But unless otherwise specified, overall threshold at stage 1 will be set to achieve 1:3 success rate (budget-wise) at stage 2.
• Common feedback to coordinators of proposals that successfully pass stage 1, but first stage ESR only sent after stage two evaluation
Specific provisions relating to two-stage
proposals
New for 2016-17
Horizon 2020:
EvaluationHow to become an evaluator
• Commission maintains a database of independent experts
• Experts called upon for the:– evaluation of proposals
– review of projects
– monitoring of programmes or policies
• Need high level expertise in research or innovation in any scientific and technological field, including managerial aspects and industry expertise
• Have at least a university degree
• Have to be available for occasional, short-term assignments
• Daily fee of approximately €450 (see model contract for details)
• List of Horizon 2020 expert evaluators available
Becoming an evaluator
• In Horizon 2020 (as of 25/8/2016):– 16 825 evaluators involved in a total of 591 927 evaluations
– 66% of the evaluators came from EU-15 countries and 15% came from EU-13 countries. 6% came from Third and Associated Countries
Evaluators
Expert evaluator background
HES REC PRC OTH PUB N/A
Gender balance amongst expert evaluators
Female Male N/A
Source European Commission Horizon 2020 Monitoring Report 2015
Horizon 2020 Evaluation
criteriaExcellence, impact and implementation
• Clarity and pertinence of the objectives;
• Soundness of the concept, and credibility of the proposed methodology;
• Extent that the proposed work is beyond the state of the art, and demonstrates innovation potential (e.g. ground-breaking objectives, novel concepts and approaches, new products, services or business and organisational models)
• Appropriate consideration of interdisciplinary approaches and, where relevant, use of stakeholder knowledge
2016-2017 Award criteria: Excellence
(RIA/IA)
(bold considered for first stage of two stage applications)
• How well does the proposal fall within the topic scope?
• How ‘frontier’ and/or ‘cutting edge’ is the research?
• How ‘innovative’ in the research?
• Will it make a significant contribution and progress existing knowledge?
• Are (new) disciplinary boundaries being crossed (where appropriate)?
Excellence
• The extent to which the outputs of the project would contribute to each of the expected impacts mentioned in the work programme under the relevant topic;
• Any substantial impacts not mentioned in the work programme, that would enhance innovation capacity, create new market opportunities, strengthen competitiveness and growth of companies, address issues related to climate change or the environment, or bring other important benefits for society.
• Quality of the proposed measures to:– Exploit and disseminate the project results (including management of
IPR), and to manage research data where relevant.
– Communicate the project activities to different target audiences
2016-2017 Award criteria: Impact (RIA/IA)
(bold considered for first stage of two stage applications)
• Meeting the Expected impacts outlined in the call topics
• Think about impact in the broadest sense:– Academic impact? Publications, conferences, data management…
– Socio-economic impact? Growth, job creation, market size, IP, monitoring of exploitation potential, policy outputs, social benefits…
– Public engagement? Communication strategy, education, media, social media, user groups…
• The extent to which project outputs will contribute at European and/or international level policy, strategies, etc.– Effectiveness of the proposed project to exploit and disseminate results
to communicate the project, and to manage research data where relevant. Open Access!
Thinking about Impact
• The Impact sections has gained more importance in Horizon 2020, as the Commission now wants to see tangible outcomes of the projects in the form on new innovations (products, services, solutions, policy recommendations, etc.)
• Furthermore, there is more focus on dissemination of results and communication of the project activities.
• Impact was the most problematic section in the first applications and many applications failed because of impact not being addressed properly.
Impact
• Most applicants state that they will reach the expected impact and be involved in a number of dissemination and exploitation activities.
• No specific details of these activities are provided and that is where the applications loose points!
• The project (incl. activities to maximise impact) must appear ready to go… at least on paper!
How to address Impact correctly?
• Quality and effectiveness of the work plan, including extent to which the resources assigned to work packages are in line with their objectives and deliverables;
• Appropriateness of the management structures and procedures, including risk and innovation management;
• Complementarity of the participants and extent to which the consortium as whole brings together the necessary expertise;
• Appropriateness of the allocation of tasks, ensuring that all participants have a valid role and adequate resources in the project to fulfil that role.
2016-2017 Award criteria: Quality and
Efficiency of the Implementation (RIA/IA)
• One of the biggest mistakes made by applicants is simply describing the consortium without showing how the various partners complement each other.
• This became even more important in the 2016-17 WP with the refined wording of the Implementation criterion:
– Complementarity of the participants and extent to which the consortium as whole brings together the necessary expertise;
– Appropriateness of the allocation of tasks, ensuring that all participants have a valid role and adequate resources in the project to fulfil that role.
Implementation
• Shorter time to grant (8 months)– affects start date of project
– proposals evaluated ‘as is’: budget should be realistic to avoid reduced implementation score
• Consortium composition:– Match with activities in the proposal
– Appropriate balance of sectors
– Commission may stipulate or offer advice on types of partner/country involvement in call text
• Previous experience managing large awards, particularly EU
• Advisory Board? Decision-making processes?
Thinking About Implementation
Other considerations
• The shortage of evaluators and massive oversubscription meant that on average evaluators had approximately 2-3 hours to evaluate proposals!
• Clear and coherent structure is necessary for the evaluators to find the most important information quickly, so follow the structure provided in the templates.
• Use charts, graphs, pictures and other visuals to help evaluators review the application properly even after the first reading – HELP your evaluator give you good score
Overall presentation of the application
• Don’t underestimate gender aspects (gender experts in all Evaluation Panels)
• Relate to EU policies on Gender Equality – cross-cutting priority in Horizon 2020
• Equal opportunities (among seconded staff and decision-makers/supervisors)
• Gender dimension in the research content (e.g. subjects or end-users)
• Gender dimension in project management and networking activities
Gender Aspects
Three gender objectives in Horizon 2020:
• Fostering gender balance in Horizon 2020 research teams
• Ensuring gender balance in decision-making
• Integrating gender/sex analysis in research and innovation content
• Gender included at application stage, implementation stage and project monitoring… Written into the Grant Agreement
Gender in Horizon 2020
Always consider SSH in H2020 projects; a
must in ‘SSH-flagged topics’
• Ethics is a consideration for all EU funded projects in all research domains
• Ethics are integral to all research, from beginning to end
• Considering ethics:
– Ensures it is within the legal framework
– Enhances the quality of research
• Strong connection between research ethics and human rights
• Ethics process for Horizon 2020 – Ethics Appraisal Procedure
Ethics in Horizon 2020
Proposals to be ‘Ethics Ready’!
• Horizon 2020 mandates open access to all scientific publications.
• There are two main options:
Open Access
Green
• Self-archiving / 'green' open access – the author, or a representative, archives the published article/final peer-reviewed manuscript in an online repository before, at the same time as, or after publication.
Gold• Open access publishing / 'gold' open access - an article
is immediately published in open access mode.
• From 2017 research data is open by default in Horizon 2020 with possibilities to opt out.– This is an expansion of the original pilot across all thematic areas of
Horizon 2020
– The Commission’s moto is ‘as open as possible as closed as necessary’
• It is possible to opt out but need a justifiable reason.
Open Research Data
Source: European Commission
• Article 29.3 of the Model Grant Agreement governs the pilot and it has been made the default going forward.
• Open research data is mentioned in the introduction to the 2016-2017 work programme.
• A new General Annex L has been added to the work programme on conditions relating to open data.
• The Commission has also updated the guidance on open research data and the emphasis on FAIR data principles has been strengthened.
Open Research Data now the default
Source: European Commission
MSCA Evaluation
process
• Proposals are allocated to one of 8 evaluation panels:
Evaluation procedure
Evaluation panels
• Chemistry (CHE)
• Social Sciences and Humanities (SOC)
• Economic Sciences (ECO)
• Information Science and Engineering (ENG)
• Environment and Geosciences (ENV)
• Life Sciences (LIF)
• Mathematics (MAT)
• Physics (PHY)
Additional multidisciplinary panels for ITNs
• European Industrial Doctorates (EID)
• European Joint Doctorates (EJD)
Additional multidisciplinary panel for IFs
• Career Re-start Panel
• Reintegration Panel
• Enterprise and Society Panel
• Evaluation scores awarded for each criteria from 0 to 5
• Each award criterion has a weighting
• Total score is subject to a threshold of 70%
Award Criteria
Excellence(50%)
Impact(30%)
Implementation(20%)
Evaluation Process
• Via Participant Portal
• Admissibility/eligibility checks1. Proposal Submission
• At least 3 evaluators (often 4)
• Individual reports produced
• Each evaluator assesses ~10 proposals for ITN, ~24 proposals for IF
2. Remote Evaluations
• Consensus reports produced
• Agreement on comments/score
• Now mostly done remotely
3. Consensus Meetings
• Lists by panel
• Projects funded in priority order until budget is exhausted4. Ranked list of proposals
Max. 5 Months to Outcome!
ERC Evaluation Process
• Excellence sole evaluation criterion
• Applied to:– the ground-breaking nature, ambition and feasibility of the research
project
– the intellectual capacity, creativity and commitment of the Principal Investigator
• Proposals marked on the above, ranging from 1 (non-competitive) to 4 (outstanding)
• Numerical marks not communicated to applicants - outcome of panel meetings expressed as A, B or C (see later).
ERC evaluation criteria
• 3 research domains, 25 panels - 2 separate sets of panel members
• Indicative budget will be allocated to each panel in proportion to the budgetary demand of its assigned proposals
• Information for Applicants document provides list of panels and keywords, indicating fields of research covered
• Lists of panel members for previous ERC calls can be found on the ERC website: http://erc.europa.eu/evaluation-panels
Peer review
Social Sciences and Humanities
Physical Sciences and Engineering
Life Sciences
• SH1: Individuals, Markets and Organisations
• SH2: Institutions, Values, Environment and Space
• SH3: The Social World, Diversity, Population
• SH4: The Human Mind and Its Complexity
• SH5: Cultures and Cultural Production
• SH6: The Study of the Human Past
• PE1: Mathematics• PE2: Fundamental
Constituents of Matter• PE3: Condensed Matter
Physics• PE4: Physical and Analytical
Chemical Sciences• PE5: Synthetic Chemistry and
Materials• PE6: Computer Science and
Informatics• PE7: Systems and
Communication Engineering• PE8: Products and Processes
Engineering• PE9: Universe Sciences• PE10: Earth System Science
• LS1: Molecular and Structural Biology and Biochemistry
• LS2: Genetics, Genomics, Bioinformatics and Systems Biology
• LS3: Cellular and Developmental Biology
• LS4: Physiology, Pathophysiology and Endocrinology
• LS5: Neurosciences and Neural Disorders
• LS6: Immunity and Infection• LS7: Diagnostics, Therapies,
Applied Medical Technology and Public Health
• LS8: Evolutionary, Population and Environmental Biology
• LS9: Applied Life Sciences and Non-Medical Biotechnology
ERC panel structure
Proposal Evaluation Process
Independent, remote reviews
by panel members
(of part B1 only)
Panel meetings and ranking
Proposals retained
for stage 2, or rejected
STEP 2 - Evaluation
Interviews of PIs (StG & CoGonly), panel meetings and
ranking
Proposals selected
Independent, remote reviews by panel members and other referees of full
proposal (parts B1 and B2)
STEP 1 - Evaluation
Eligibility check
• Step 1 (Part B1 of proposal)– A: is of sufficient quality to pass to Step 2 of the evaluation
– B: is of high quality but not sufficient to pass to Step 2 of the evaluation
– C: is not of sufficient quality to pass to Step 2 of the evaluation
Applicants scoring B or C told the ranking range of their proposal out of those evaluated by the panel
• Step 2 (full proposal and interview for StG and CoG)– A: fully meets the ERC's excellence criterion and is recommended for
funding if sufficient funds are available
– B: meets some but not all elements of the ERC's excellence criterion and will not be funded
Applicants told the ranking range of their proposal out of the proposals evaluated by the panel
Outcome of evaluation
Proportions Per Score (From 2015 Calls)
Evaluated step 2 StG2015 CoG2015 AdG2015
All UK All UK All UK
A funded 46.6% 40.7% 43.7% 44.0% 42.1% 44.0%
A not funded 16.8% 10.7% 20.2% 20.0% 13.1% 14.0%
B 36.6% 48.6% 36.1% 36.0% 44.8% 42.0%
Evaluated step 1 StG2015 CoG2015 AdG2015
All UK All UK All UKA (through to step 2) 26.2% 30.7% 34.5% 41.9% 34.1% 42.3%
B 44.2% 49.5% 34.6% 36.3% 36.7% 40.7%
C 29.6% 19.8% 30.9% 21.8% 29.2% 17.0%
Further Information
• Horizon 2020 Grants Manual:http://ec.europa.eu/research/participants/portal/desktop/en/funding/reference_docs.html#h2020-grants-manual– Section on proposal submission and evaluation
– Guidance on the evaluation of some H2020 aspects (e.g. innovation, social sciences and humanities)
• Horizon 2020 Online Manual: Section on evaluation of proposals:http://ec.europa.eu/research/participants/docs/h2020-funding-guide/grants/from-evaluation-to-grant-signature/evaluation-of-proposals_en.htm
• Work Programme 2016-2017: General Annexes:http://ec.europa.eu/research/participants/data/ref/h2020/other/wp/2016-2017/annexes/h2020-wp1617-annex-ga_en.pdf
Useful links
Commission briefings for evaluators
http://ec.europa.eu/research/participants/data/support/expert/h2020_expert-briefing_en.pdf
http://ec.europa.eu/research/participants/data/ref/h2020/grants_manual/pse/h2020-evaluation-faq_en.pdf
• Gendered Innovation, Stanford University project: https://genderedinnovations.stanford.edu/
- practical tools for researchers: methods to be used in a research project; case studies; checklist
• Horizon 2020 Manual, part on Gender equality: http://ec.europa.eu/research/participants/docs/h2020-funding-guide/cross-cutting-issues/gender_en.htm
Gender Aspects - Links
• H2020 Gender Advisory Group paper on preparing grants that integrate the gender dimension into research. http://ec.europa.eu/transparency/regexpert/index.cfm?do=groupDetail.groupDetailDoc&id=18892&no=1
• https://ec.europa.eu/programmes/horizon2020/en/h2020-section/responsible-research-innovation
• Rome Declaration on Responsible Research and Innovation in Europe, November 2014https://ec.europa.eu/research/swafs/pdf/rome_declaration_RRI_final_21_November.pdf
• Report from the Expert Group on Policy Indicators for Responsible Research and Innovationhttp://ec.europa.eu/research/swafs/pdf/pub_rri/rri_indicators_final_version.pdf
• Open Science: https://ec.europa.eu/digital-agenda/en/open-science
Responsible Research and Innovation