20201015 schofield kth evaluators view public
TRANSCRIPT
14.10.2020
1
KTH HE Week 2020-10-15
THE EVALUATOR’S VIEW:HOW TO WRITE A COMPETITIVE
EU PROPOSAL
Monica SchofieldTutech Innovation
KTH Horizon Europe Week 15 October 2020
These slides are for personal use of participants only
tutech.deKTH HE Week - 2020-10-15
Today’s topics
• Some words on transition H2020-HE
• The evaluation criteria and process
• Who evaluates and how
• Getting your message across
• Addressing pathways to impact
• Forming the optimal partnership
• Checking your network
Ask questions whenever
Time limit!
KTH HE Week 2020-10-15
HORIZON 2020 HORIZON EUROPE
14.10.2020
2
tutech.deKTH HE Week - 2020-10-15
Framework 9 – Horizon Europe
STILL ….
tutech.deKTH HE Week - 2020-10-15
plus c'est la même chose
• A Programme
• Work Programme
• Calls
• Submission of proposals
• Principle of partnerships
• Evaluation criteria (largely)
• Funding rates (largely)
Continuity in much of theoperational aspects(as requested by the community)
tutech.deKTH HE Week - 2020-10-15
Change lies in the political context…
tutech.deKTH HE Week - 2020-10-15
The politics of 2019-2020 (COM slide Feb 2020)
Preparation of HE started2016/17.
2019 New Commission,New Parliament
2020 Brexit
14.10.2020
3
tutech.deKTH HE Week - 2020-10-15
Political message: R&I is needed to deliver EU policy objectives
A European Green Deal
An economy that works for people
A Europe fit for the digital age
Protecting our European way of life
A stronger Europe in the world
A new push for European democracy
tutech.deKTH HE Week - 2020-10-15
The politics of 2020
tutech.deKTH HE Week - 2020-10-15
“strengthen the impact of research and innovation in developing, supporting and implementing Union policies and … to address global challenges, including climate change and the Sustainable Development Goals.”
tutech.deKTH HE Week - 2020-10-15
The new priority – TAKE NOTE
14.10.2020
4
tutech.deKTH HE Week - 2020-10-15
Evaluation – principles unchanged
• Continuity with H2020
• Aim to improve feedback toproposers
• Aim to improve transparency androbustness
• Possibility that questions may be put to proposers
• Piloting of anonymous stage 1 proposals
S&T Excellence
ImpactImpact
Implementation
0…5
0…5
0…5
Ref: R&I Day session: Quality, transparency and fairness in Horizon Europe proposal evaluation’ 22.9.2020
tutech.de
Excellence
https://ec.europa.eu/research/participants/data/ref/h2020/call_ptef/ef/2018-2020/h2020-call-ef-ria-ia-csa-2018-20_en.pdf
Award criteria H2020 RIA/IA
Clarity and pertinence of the objectives
Soundness of the concept, and credibility of the proposed methodology
Extent that the proposed work is beyond the state of the art, and demonstrates innovation potential (e.g. ground-breaking objectives, novel concepts and approaches, new products, services or business and organisational models)
Appropriate consideration of interdisciplinary approaches and, where relevant, use of stakeholder knowledge and gender dimension in research and innovation.
WHAT
tutech.de
Impact
https://ec.europa.eu/research/participants/data/ref/h2020/call_ptef/ef/2018-2020/h2020-call-ef-ria-ia-csa-2018-20_en.pdf
Award criteria H2020 RIA/IA
The extent to which the outputs of the project would contribute to each of the expected impacts mentioned in the work programme under the relevant topic;
Any substantial impacts not mentioned in the work programme, that would enhance innovation capacity, create new market opportunities, strengthen competitiveness and growth of companies, address issues related to climate change or the environment, or bring other important benefits for society;
Quality of the proposed measures to: Exploit and disseminate the project results (including management of IPR), and to manage research data where relevant.Communicate the project activities to different target audiences
WHY
tutech.de
Implementation
https://ec.europa.eu/research/participants/data/ref/h2020/call_ptef/ef/2018-2020/h2020-call-ef-ria-ia-csa-2018-20_en.pdf
Award criteria H2020 RIA/IA
• Quality and effectiveness of the work plan, including extent to which the resources assigned to work packages are in line with their objectives and deliverables;
• Appropriateness of the management structures and procedures, including risk and innovation management;
• Complementarity of the participants and extent to which the consortium as whole brings together the necessary expertise;
• Appropriateness of the allocation of tasks, ensuring that all participants have a valid role and adequate resources in the project to fulfil that role.
HOW
14.10.2020
5
tutech.deKTH HE Week - 2020-10-15
Sub-criteria likely tochange and may even varyacross the calls
tutech.deKTH HE Week - 2020-10-15
Horizon EuropeExcellentScience
IndustrialLeadership
LEIT
SocietalChallenges
tutech.deKTH HE Week - 2020-10-15
Evaluation criteria
Continuity with H2020
tutech.deKTH HE Week - 2020-10-15
Preparing a proposal
The Call
Ideas
Partners
The document
that sells the project
14.10.2020
6
tutech.deKTH HE Week - 2020-10-15
Receipt of proposalsReceipt of proposals
IndividualevaluationIndividualevaluation
Consensusgroup
Consensusgroup
Panel ReviewPanel Review
FinalisationFinalisation
Evaluators
IndividualEvaluationReports
(Usually done
remotely)
ConsensusReport
(May be done remotely)
Panel report
Evaluation Summary Report
Panel ranked list
Eligibility check
Allocation of proposals to evaluators
Final ranked list
Overview of the evaluation process
Possibility to ask questions of proposers
Portfolio assessment
tutech.deKTH HE Week - 2020-10-15
Process as seen by the evaluator
• Briefing
• Access to proposals
• Check if conflict of interest
• Briefing meeting by webinar
• Reading time (half a day per (full) IA,RIA proposal)
• Entry of individual scores and comments into evaluation tool
• No visibility on other scores until all evaluators scored• Scores must be justified by comments (“comments first, score second”)
• Consensus meeting
• (Ranking)
tutech.deKTH HE Week - 2020-10-15
Proposal evaluation
Ideas
Partners
The Call
The document that says it
all!
tutech.deKTH HE Week - 2020-10-15
Before you start, look at the criteria for selection
Guide for applicants
Work programme and its context
(Destination)
AND check what has already been funded!
14.10.2020
7
KTH HE Week 2020-10-15
ADDRESSING THE EVALUATORS
tutech.deKTH HE Week - 2020-10-15
Where do evaluators come from?
• Named in the expert database
• Known to Project Officers
• Competence in an aspect of the call (Not just specialists in the science!)
• Gender balance
• Nationality balance
• Industry/academia (other) balance
• Newcomers vs old timers
• Available
tutech.deKTH HE Week - 2020-10-15
What this means in practice
Challenge to recruit people from a range of organisations
Payment of 450 euros per day - filters out those who can earnmore by doing other things…
… unless they are motivated for other reasons
Taken as a group, the evaluators are pretty competent, but theywon‘t all be deep specialists in your field
tutech.deKTH HE Week - 2020-10-15
Stating the obvious
Evaluators are human – this is not an infallible process
14.10.2020
8
tutech.deKTH HE Week - 2020-10-15
http://ec.europa.eu/research/participants/portal/desktop/en/experts
Become an evaluator!
tutech.deKTH HE Week - 2020-10-15
How do evaluators read proposals?
• Not necessarily from beginning to end
• May read a couple of proposals before scoring
• May pick the easiest to judge first!
• May read proposals on the move
tutech.deKTH HE Week - 2020-10-15
Keeping evaluators well disposed
• Make the document easy to read
• Don‘t cram pages and make sure diagram legends can be read on a tablet
• Write so it is interesting and fluid to read
• Avoid “motherhood and apple pie“ statements• Avoid repeating chunks of the call• Don‘t write in policy speak clichés
• Don‘t say you are fantastic, unique etc, but make clear what makes this a special project
tutech.deKTH HE Week - 2020-10-15
Winning proposals
• Original or creative project idea – EXCITING!
• Well articulated (measurable) objectives – CLEAR
• A well-matched partnership of different kinds of skill – GREAT PROJECT COMMUNITY
• Impacts stated in call are fully addressed – RELATES TO NEEDS
• Evidence of a strategy to move knowledge beyond the researchcommunity – SOCIETY WILL REALLY BENEFIT
• The implementation plan is thought through – CAN DELIVER
• The proposal is reasonably priced – SO THAT WE CAN FUND MORE PROJECTS
14.10.2020
9
tutech.deKTH HE Week - 2020-10-15
Section 1What do you want to achieve and how?
Section 1What do you want to achieve and how?
tutech.deKTH HE Week - 2020-10-15
What the evaluators take from section 1
• What is the project trying to achieve and how it relates to the call?
• What makes this project special?
• What is the key result going to be?
• Why should I believe you can do this?
• Where does this project fit into previous work?
• Does the partnership look as though it knows what it wants to achievetogether?
tutech.de
Proposal Section 1 – objectives
OPENING STATEMENT: MAKE IT EASY FOR THE EVALUATOR TO UNDERSTAND THE PROJECT
• WHAT is the project aimed at achieving?
• WHY is this relevant to the ‘bigger picture’?
• WHAT will be the result at the end of the project?
The specific objectives of your project shouldinclude the objective to achieve impact.
tutech.deKTH HE Week - 2020-10-15
Be specific
This project
• Will develop X,Y, Z
• Will provide a means to
• Will achieve a something
NOT
• Will bring together lots of people
• Will analyse lots of things
• Will gather lots of data
14.10.2020
10
tutech.deKTH HE Week - 2020-10-15
Positioning your project
• Be specific about the context of your project
• Make the distinguishing features very clear
• Don‘t confuse the evaluators or give the impression youare repeating work done before
• As a rule, evaluators don‘t appreciate projects that aresimply a continuation of a previous project
What would the world miss of significant value if this project was not funded?
tutech.deKTH HE Week - 2020-10-15
Section 2Why is this worth doing and
what are you going to do to ensure the knowledge is shared and used?
Section 2Why is this worth doing and
what are you going to do to ensure the knowledge is shared and used?
tutech.deKTH HE Week - 2020-10-15
ResearchersResearchers
DevelopersDevelopers
Users, society Users, society
Linking knowledge to impact
PAVING THE PATHWAYS TO IMPACTMONITORING AND EVALUATING IMPACT IN
HORIZON EUROPE
14.10.2020
11
TRACKING IMPACT IN HORIZON EUROPE
Scientific impact Promote scientific excellence, support the creation and diffusion of high-quality new fundamental and applied knowledge, skills, training
and mobility of researchers, attract talent at all levels, and contribute to full engagement of Union's talent pool in actions supported under the Programme.
Societal impactGenerate knowledge, strengthen the impact of R&I in developing, supporting and implementing Union policies, and support the uptake
of innovative solutions in industry, notably in SMEs, and society to address global challenges, inter alia the SDGs
Economic impact Foster all forms of innovation, facilitate technological development, demonstration and knowledge transfer, and strengthen deployment of
innovative solutions
THREE TYPES OF IMPACT BASED ON OBJECTIVES
tutech.deKTH HE Week - 2020-10-15
Software technologies
Communication & networking
Micro, nano& opto
electronics
Micro & nanosystems
InterfacesKnowledge technologies, digital content
Miniaturised, low cost, low power
components and microsystems
Pervasive, mobile, wireless, trustful infrastructures
Natural interactions with “knowledge”
Applied IT for major societal & economic challenges
Trust & security
Complexproblem solving
IT for economic challenges
IT for societal challenges
Addressing pathways – link to societal needs
The target: enabling anywhere, any time natural and enjoyable access to IT services for ALL
tutech.deKTH HE Week - 2020-10-15
DESTINATION: Climate neutral, circular and digitised production
Breakthrough science New technology
Societal and economic factors
Regulation
Putting research into a bigger context
Changing
consumer
behaviour
New systems
tutech.deKTH HE Week - 2020-10-15
HE Work Programme structure
Context
Destination
Expected impact
Scope - topics
Conditions
The overall ambitions
What is ultimately to be achieved
What the project should lead to
Focus of the project
Budget, Partners …
14.10.2020
12
tutech.deKTH HE Week - 2020-10-15
Addressing impact
What are you going to do in the projectto support this happening
- pathways to impact, dissemination
Why should I believe you will actuallyaddress this?
- choice of partners, deliverables
In what way could your project have impact…- narrative, link to call
tutech.deKTH HE Week - 2020-10-15
Think beyond the project
Technological
opportunity -
breakthrough
Formation of
products or
services
Public/Market/
Regulatory
acceptance
Project content Innovation scenario
Influence on
public policyPolitical
/societal
acceptance
Utilisation by society
Development
of a challenge
solution
tutech.deKTH HE Week - 2020-10-15
Think about the “Pathway to Impact”
• What can you do in the project?
• Who should you involve to ensure the outputs of the project are transported beyond the project?
• What is the best way to involve them?
tutech.deKTH HE Week - 2020-10-15
Reading the WP: example outcome
TWIN-TRANSITION-23-2021: The use of BIM for circular construction by valorisation of construction and demolition waste (RIA)
Expected outcomes: Projects are expected to contribute to the following outcomes:
• Reduce the quantity of construction and demolition waste (CDW)
• Demonstrate a more circular and climate neutral construction sector
• Increase the reusability potential of construction products post demolition and reduce the downcycling of construction waste by facilitating modular dismantling of complex construction products
14.10.2020
13
tutech.deKTH HE Week - 2020-10-15
Reading the WP: example scope
Scope: Based on volume, construction and demolition waste (CDW) is the largest waste stream in the EU. Considering that most of the waste quantities is concrete and steel, the embodied energy and CO2 emission in the CDW is significant. By recycling and reusing CDW in new constructions, the sector would come closer to the targets of becoming fully circular and climate neutral.
Quantitative waste estimation is crucial for waste management. This could be achieved by utilising technologies such as Building Information Modelling (BIM). Such a model could provide information of the materials type (e.g. whether there are hazardous materials that require special care) and quantities, and thus an estimation of the duration for removal, the number of pick-up trucks needed, cost, etc. and make waste separation easier and faster, e.g. by combining BIM with robotic arms.
tutech.deKTH HE Week - 2020-10-15
Reading the WP: example scope
Proposals should:
• Develop, test and promote the necessary BIM applications for CDW management in different types of constructions;
• Develop automated solutions for waste separation process;
• Produce all required training material for the proper use of the developed technologies;
• Demonstrate of all developed solutions in at least four construction sites across different Member States;
• Assess the value of the solutions in terms of the additional monetary value/reduction of CO2 emissions produced
• Finally, the projects should provide contributions to relevant standards.
tutech.deKTH HE Week - 2020-10-15
Reading the WP – start with the outcome
Expected outcome Howwill this be achieved? Who has to be involved? How toconnect them to the project? How can you measure/predict theoutcome? PARTNERS NEEDED FOR OUTCOMES
Reduce the quantity of construction and demolition waste
Demonstrate a more circular and climate neutral construction sector
Increase the reusability potential of construction products post demolition and reduce the downcyclingof construction waste by facilitating modular dismantling of complex construction products
tutech.deKTH HE Week - 2020-10-15
Reading the WP – start with the outcome
Scope What competencies are needed to cover scope? WHAT PARTNERS NEEDED TO COVER SCOPE
Develop, test and promote the necessary BIM applications ….
Develop automated solutions for waste separation process
Produce all required training material ….
Demonstrate developed solutions in at least four construction sites …
Assess the value of the solutions in terms of the additional monetary value/reduction of CO2 emissions produced
Contribute to standards
14.10.2020
14
tutech.de
Developing a strategic approach to dissemination
tutech.deKTH HE Week - 2020-10-15
Dissemination – to support impact
Target groups –who to inform
Content – what information to provide
Channels – how best to reach target
ResearchersThe publicPolicy makersRegulatorsBusiness …
ResultsIdeasProblemsPR…
WebinarsNewsletterConferencesSocial media…
Which policy makers, what results, which conferences?Why these?
tutech.deKTH HE Week - 2020-10-15
Who might have an interest in the results?
• Colleagues – researchers in same field
• Researchers from other disciplines (why?)
• Policy makers
• Regulators, standards bodies
• Government
• Local authorities
• General public
• Industry
• NGOs, civil society
• The media, creative industry
• Funding bodies! (e.g. for new programmes)
• ….tutech.deKTH HE Week - 2020-10-15
Involving people
To achieve the specified impacts who needs to be involved?
• Users• Policy makers• Opinion formers
What is the best way to involve them?
• As partners• As advisors• As target for knowledge transfer activities
14.10.2020
15
KTH HE Week 2020-10-15
THE PARTNERSHIP
tutech.deKTH HE Week - 2020-10-15
Align the outputs to what you say you will do
Excellent science
Industrial leadership
Societalchallenges
• Shift frontiers of knowledge• Publish in top journals
• Achieve innovation• Achieve breakthrough products• …
• Contribute to transformations in society
Evaluators will look for evidence of ability to ...Project Focus
tutech.deKTH HE Week - 2020-10-15
Align the partnership with the stated goal
Excellent science
Industrial leadership
Societalchallenges
• Top class complementaryresearch partners
• Industrial commitment
• Partners capable of implementingand duplicating
What evaluators will look for in partners
tutech.deKTH HE Week - 2020-10-15
Form partnerships that support the project logic
Achieve a breakthrough – a completely new way of doing things
Move technology towards widespread application
Address the needs of a specific sector
Provide support for the implementation/acceptance of
aspects of policies or programmes
Bring communities of experts or practitioners together
Do you have the top guys in the project?Can you change the mindset of the research community?
Knowledge of policies?Involvement in key committees?Track record in giving policy advice?
Construction of network?Sustainability of network?
User involvement?How to connect to the status quo?Routes to (industrial) exploitation?
Are the “standard bearers” involved?Do you show you know the sector?
Funding for a research facility Funding in the longtermAccess to the facility?
Funding to support career development of young researchers
How and how many will it help?
14.10.2020
16
tutech.deKTH HE Week - 2020-10-15
The partnership should reflect the TRL
PAGE
61
Full deployment
First of the kind commercial system (commercial level)
Demonstration System (pre-commercial scale)
Technology demonstrated
Technology validated in environment
Technology validated in lab
Experimental proof of concept
Technology concept formulated
Basic principles observed
Claudia Häfner24.09.2013 KIT, Karlsruhe
TRL 1:
TRL 2:
TRL 3:
TRL 4:
TRL 5:
TRL 6:
TRL 7:
TRL 8:
TRL 9:
tutech.deKTH HE Week - 2020-10-15
Balance what partners represent
Partner a
Spec. Gen. Exploit. User SME Region Branch ...
Partner b
Partner d
Partner c
Partner e
Partner f
tutech.deKTH HE Week - 2020-10-15
Do this exercise before you invite too many of your research friends to join the partnership!
tutech.deKTH HE Week - 2020-10-15
14.10.2020
17
tutech.deKTH HE Week - 2020-10-15
How good is your network?
SE EU
Researchers
SMEs
Big industry
Policy advisers
Users
….
tutech.deKTH HE Week - 2020-10-15
Stating the obvious
If the budgets of proposals exceed thefunding budgets, not all proposals can win funding, however good they are!
Success rates:FP7 – 19%H2020 - 12.6%HE ?
tutech.deKTH HE Week - 2020-10-15
Worthwhile anyway?
“Even if the project isn’t funded, this has been worthwhile for the partners I have got to know, and the process has helped me structure ideasfor my future research directions”
Monica Schofield
THANK YOU!