QualitativeEvaluationofcompletedprojectsfundedbytheEuropeanResearchCouncil

July2016

1

Introduction

The European Research Council (ERC) was established in 2007 with the goal ofproviding funding to investigators to pursue ground‐breaking, high‐risk/high‐gainresearch in Europe. It is one of the premier research funders in the EuropeanUnion.ScientistsandscholarsofanynationalityandinanydisciplinecanapplytotheERCforagrant to undertake research at the frontiers of knowledge, free of any thematicconstraints, in an EU Member State or an Associated Country. Researchers at acomparablecareerstagecompeteforStarting,ConsolidatororAdvancedGrants—thesolecriterionforreceivingresearchfundingisscientificqualityaimingatexcellenceofboththeprojectandtheprincipalinvestigator.Theevaluationofproposalsisconductedbymeans of a structure of high‐level peer review panels; the ERC currently has 251panels coveringall fieldsacross threeresearchdomains:LifeSciences (9panels,LS1–LS9),PhysicalSciencesandEngineering(10panels,PE1–PE10),andSocialSciencesandHumanities (6 panels, SH1–SH6) (Table 1). Following the publication of a call,researchers can submit their proposals in a ‘bottom up’manner. The panel structureserves solely as an organisational tool to gather experts with appropriate collectiveexpertise.

Since 2007, more than 60000 project proposals have been submitted to the ERC, ofwhich,todate,some6000havebeenselectedforfunding,representinganinvestmentof9.8billion euros. At the end of 2014, more than 500 ERC‐funded projects had beenfinalised. The vast majority of ERC‐funded projects are still ongoing as they wereinitiatedlater.

When public funds are disbursed to support research, especially on this scale, it isimportanttoevaluatewhetherthegoalsetfortheprogrammeisbeingmetandwhetherthepeerreviewproceduresinplaceleadtothatground‐breakingresearchprojectsarefunded.Furthermore,astheERCfundsprojectsinastrictly ‘bottom‐up’approach, it isalso of interest to evaluate the potential for societal impact of the outcome of theprojects funded, even though it should be noted that societal impact often onlymaterialisesmanyyearsafteraprojectiscompleted.

There are various indicators that can be used to evaluate the outcomes of researchprojects,includingscientificimpactanddevelopments,bibliometricanalysisofscientificpublications,and,dependingonthedomains,generationofpatentsandestablishmentofspin‐off companies. In 2014, the ERC conducted a citation analysis of the 30319publications retrieved from Thomson Reuters’ Web of Science database as havingacknowledgedERCfunding(asofSeptember2014).Overall,2005articlesandreviewsacknowledgingERCsupport(correspondingto7%ofthesepublications)wereclassifiedinthetop1%ofmosthighlycitedpublicationsintheirscientificdisciplineandyearofpublication. This analysis showed that research funded by the ERC has a scientificimpactfaraboveaverage.

1In2007,theERCpanelstructureconsistedof20panels,butin2008thenumberofpanelswasincreasedto25andthisstructurehasremainedstablesincethen.

2

Qualitativepeerreviewanalysisremains thegoldstandard for in‐depthassessmentofresearchoutcomes.ThisiswhytheERCScientificCouncilrequested,aspartofthe2015Work Programme2, an analysis of the results and outcomes of ERC research fundingevaluation,followingthisapproach.

Inthisevaluation,whichservesasapilotexerciseforthefutureevaluationofcompletedERC‐fundedprojects, thequalitativeevaluationof199completedERC‐fundedprojectswas undertaken by independent high‐level scientists who were selected by the ERCScientificCouncil.Theaimofthisdocumentistoreportontheoveralloutcomeofthisevaluation.

Thecompletedprojectsevaluatedwerefundedinthe frameworkof theStartingGrantand Advanced Grant funding schemes that the ERC implemented in its first years ofexistence.TheStartingGrantschemeprovidedfundsofupto1.5millioneurostojuniorprincipal investigators (2 to 9 years after PhD) for projects of up to five years induration,while the Advanced Grant scheme focused on senior principal investigatorswho received funding of up to 2.5 million euros for projects of up to five years induration.TheConsolidatorGrantschemewasonlyintroducedin2013and,thus,therearenotyetanycompletedprojectsforthisgranttype.

Table1:TheERCpanelstructure

Panel PaneltitleLifeSciencesLS1 MolecularandStructuralBiologyandBiochemistryLS2 Genetics,Genomics,BioinformaticsandSystemsBiologyLS3 CellularandDevelopmentalBiologyLS4 Physiology,PathophysiologyandEndocrinologyLS5 NeurosciencesandNeuralDisordersLS6 ImmunityandInfectionLS7 DiagnosticTools,TherapiesandPublicHealthLS8 Evolutionary,PopulationandEnvironmentalBiologyLS9 AppliedLifeSciencesandNon‐MedicalBiotechnologyPhysicalSciencesandEngineeringPE1 MathematicsPE2 FundamentalConstituentsofMatterPE3 CondensedMatterPhysicsPE4 PhysicalandAnalyticalChemicalSciencesPE5 SyntheticChemistryandMaterialsPE6 ComputerScienceandInformaticsPE7 SystemsandCommunicationEngineeringPE8 ProductsandProcessesEngineeringPE9 UniverseSciencesPE10 EarthSystemScienceSocialSciencesandHumanities

2https://erc.europa.eu/sites/default/files/document/file/ERC_Work_Programme_2015.pdf

3

SH1 Individuals,InstitutionsandMarketsSH2 Institutions,Values,BeliefsandBehaviourSH3 Environment,SpaceandPopulationSH4 TheHumanMindandItsComplexitySH5 CulturesandCulturalProductionSH6 TheStudyoftheHumanPast

Methodology

The ERC followed a common approach to evaluation based on peer review for allcompletedprojects. 25 evaluationpanels, corresponding to the25 scientific panels oftheERC,wereformed,eachcomposedofthreeexperts;twowithexperienceasanERCpanelmember or panel chair, and onewithout any previous participation on an ERCevaluationpanel.Members of the selectionpanels thatmade the decision to fund theprojectstobeevaluatedwerenotrecruitedtotheseevaluationpanels.Eachevaluationpanelevaluated8projects3;5StartingGrantprojectsand3AdvancedGrant projects, reflecting the ERC distribution of funding between younger andmoreexperiencedresearchers.MostoftheprojectswerefundedinthefirsttwoERCcalls(i.e.in2007and2008).Theprojectstobeevaluatedwereselectedbasedontheprojectstartdate,andwereallocatedtothe25panelsaccordingtothe2013panelidentifiersoftheERC,i.e.thelastpanelstructureusedintheSeventhFrameworkProgramme(FP7).Eachprojectwas evaluated by 2members of the evaluation panel and, if necessary, by anadditional remote reviewer. These additional reviewers were appointed by themembersoftheevaluationpanel.Onepanelmemberwasappointedasleadreviewerforeachproject.Aproceduretodetectconflictsofinterestandprotecttheconfidentialityoftheexercisewasestablished,andtheevaluatorsreceivedanhonorariumfortheirwork.

Theevaluatorswereprovidedwiththefollowingmaterial:(i) TheDescription ofWork – adocumentbased on theproject proposal that

motivated the selection for funding and which was part of the GrantAgreement;

(ii) TheFinalScientificReportsubmittedbytheprincipalinvestigatorattheendof the project that includes a self‐assessment describing the projectachievements, and information on project outputs such as publications,awardsandpatents;

(iii) Apublicationlistandbibliometricanalysis;

(iv) Whereapplicable, informationonanyProofofConceptGrants (ERCgrantsthataredesignedtobridgethegapbetweenresearchandtheearlieststageofmarketableinnovation)associatedwiththeproject.

Inaddition,theevaluatorswereencouragedtoconsideranyotherinformationpubliclyavailable throughonlineresources.Notably, theevaluatorswereencouragedtoassessanypublicationby theprincipal investigatorspublishedafter theendof theproject ifERC fundingwas acknowledged or if itwas deemed to be directly linked to the ERC‐3Oneevaluationpanelevaluated7projects.

4

fundedproject.

Theevaluatorswereaskedtousetheirprofessionaljudgementtoformanoverallviewof the scientific quality of the project achievements. Each evaluator delivered onewrittenreviewofeachevaluatedproject,andthepanelasawholedraftedaconsensusreportforeachofthe8projectsevaluatedbythepanel.Theprimaryfocusoftheprojectreviewswastohighlightthefrontiernatureoftheresults,includinganybreakthroughsor important scientific advancesof knowledge.Theproject reviewsconsistedof threeparts:

(i) Abrief questionnaire addressing the scientific advancesmade, the level ofinterdisciplinarity,andimpactoutsidethescientificdomain(ifapplicable);

(ii) Areviewtextdescribingandassessingtheprojectfindings;

(iii) Anoverallgradebasedonthescientificresults.

The overall grades used for the evaluation panels to categorise the projects weredefinedasfollows:

(A)‐scientificbreakthrough;(B)‐majorscientificadvance;(C)‐incrementalscientificcontribution;(D)‐noappreciablescientificcontribution.

In the first phase of the evaluation, the panels were given access to the evaluationdocuments. Subsequently, the panels met for a one‐day meeting in Brussels in June2015. Themeeting consisted of an introduction to the exercise and a discussion in aplenarysession,followedbyindividualbreak‐outsessionsforeachpaneltoprepareandinitiatetheevaluationandnominateremotereviewerswhereadditionalexpertisewasneeded. In the second phase, the evaluators remotely prepared individual reviews.Followingthis,virtualpanelmeetingswereheldbyvideoconferenceinwhichthepanelsdiscussed the reviews and agreed on the main points for the panel’s consensusassessment. Finally, the panels prepared one consolidated review report for eachproject,andapanelreportsummarisingageneralviewontheadvancesandimpactoftheresearchfundedinthefieldofthepanel.Findings

Overalloutcome

The evaluation panels identified 43 projects (ca. 21%) as having led to a scientificbreakthrough (gradeA) and 99 projects (ca. 50%) as having led to amajor scientificadvance(gradeB).Takentogether,ca.71%oftheevaluatedprojectswereassessedashavingledtoamajorscientificadvanceorascientificbreakthrough(Figure1),i.e.highlysuccessfulprojects.Theexercisealsoshowedthat50projects–thatis25%‐deliveredanincrementalscientificcontribution,andinafewcases‐ca.4%‐hadnoappreciablescientificoutput.Itshouldbenotedthoughthattheselattercategoriescontainprojectswhichmay not have led to the intended outcomes due to the high risk nature of theprojects,aswellasprojectswhichdeliveredratherpooroutputs.Itisimportanttokeepin mind that the small sample size limits the conclusions that can be drawn. TheseresultscannotnecessarilybeextrapolatedtothefullsetofERC‐fundedprojects.

5

Figure1:Overallgradeattributedtoprojects.Outcomebygranttype

AhigherproportionoftheAdvancedGrantsevaluatedinthisexercisewereawardedagradeAthanStartingGrants(Figure2andTable2).WhengradesAandBarecombined,however, a rather similar picture emerges for Starting Grants and Advanced Grants.Therearenoclearindicationsofdifferencesbetweenthetwogranttypesintermsofthequalityoftheresultsobtained.

Figure2:Overallgradeattributedtoprojectsbygranttype.StG:StartingGrant;AdG:AdvancedGrant.

Table2:Overallgradeattributedtoprojectsbygranttype. A‐Scientific

breakthroughB‐Majorscientificadvance

C‐Incrementalscientificcontribution

D‐ Noappreciablescientificcontribution

Total

StartingGrant

21 67 30 3 121

AdvancedGrant

22 32 20 4 78

Total 43 99 50 7 199Scientificimpactandrisk

Theoverall scientific impactof these199researchprojects fundedby theERC isveryhigh.Evenprojectscategorisedasincrementalintheoverallassessmentcouldresultinsignificant scientific advances or open up new research avenues. The reviews of theprojects show that, in general, funded projects were ambitious and have led to newimportant results being published in key journals. For many projects, these outputswere considered to have had a major impact, as indicated by the high numbers ofcitationscollected.

6

Although this is not directly related to the scientific quality of the project'sachievements, another feature common to many of the funded projects is theconsolidationoftheresearchgroupsoftheprincipalinvestigators.ERCfundingenablesprincipal investigators to createor consolidate their researchgroup; thiswaspointedoutbyreviewersasaverypositiveaspectduetothehighimpactonthecareerofearly‐stage researcherswithin the funded research team.Many principal investigators alsoreceivedasignificantcareerboostduringtheimplementationoftheirERCgrant.ThisishighlightedbymanyreviewersasamajorimpactofERCfunding.

Projects that were graded as incremental or not providing appreciable scientificcontributions could be an indication of the risk taken by the evaluation panelswhendecidingontheproposalstobefunded.Theidentificationofresearchproposalswiththepotential to lead to breakthrough results is a very challenging endeavour: at theselectionstage,evaluationpanelsareaskedtoidentifythemostpromisingamongstthemanybrilliantideasputforwardbyapplicantsandthishasahighintrinsicrisk.Panelsareguidedtotaketheseriskswhileaimingforhigh‐gainresearchprojects.Theresultsobtainedinthisexerciseprovidesomeindicationsthatthisguidancewasfollowedtoalargeextent.

The reviewerspointedout thatmanyof theprojects given the two lowest gradesdidachieve some of their goals, or partially delivered on their main aim. For example,severalsuchprojectssucceededindevelopingnovelmethodologyoraninstrument,buthave not (yet) succeeded in answering the planned scientific questions by using thismethodology/instrument. In contrast, for someof theprojects in these categories, theevaluationhighlightedtheirmodestpublicationoutputs, limited impactonthe fieldorcriticised the project design. It is important to highlight that the snapshot of thescientificimpactoftheprojectsevaluatedisfromashorttimeaftertheircompletion.Inmany areas, scientific impact takes longer to become apparent and it would beinterestingtocomparethecurrentfindingstothosethatwouldemergebyredoingthisevaluationinafewyearsfromnow.

Thedistributionofthelevelofscientificimpactfoundfitswellwiththeexpectedpatternforafrontierresearchfundingscheme.Thedistributionpeakedaroundprojectsleadingto major scientific advances with a moderate tail of projects with no appreciablescientificcontributions,thusreflectingthebalancebetweenhighriskandhighgainthatguidedthedecisionoftheERCselectionpanelsthatevaluatedtheproposals.Interdisciplinarity

The reviewers also indicated that many of the projects evaluated have an importantinterdisciplinary component. Thiswas linked to the fact that the research performedfoundrecognitionorapplicabilityoutsideitsmainfield,orbecausetheresearchbroughttogetherareasthatpreviouslydidnothavemanyinteractions(Figure3andFigure4).

7

Figure3:Recognitionorapplicabilityoftheresearchperformedoutsideitsmainfield.

Figure4:Researchbringingtogetherareasthatpreviouslydidnothavemuchinteraction.Impact(e.g.oneconomy,onsociety,onpolicy‐making)inadditiontoscientificimpact

Although it is still very early to consider the long‐term impact of the projects, theevaluatorswere asked to assess the impact (e.g. on the economy, society and policy‐making)ofthefundedprojectsinadditiontotheirscientificimpact.Theevaluatorswereasked two questions: (i) concerning impact that is already apparent, and (ii) thepotentialforimpactinthefuture:

(i) Inadditiontoitsscientificimpact,towhatextenthastheprojecthadothertypesofimpact(e.g.,ontheeconomy,societyandpolicy‐making)?

(ii) In addition to its scientific impact, in your opinion, could the project haveothertypesofimpact(e.g.,oneconomy,onsociety,onpolicy‐making)inthefuture?

TheresultsareshowninFigure5andFigure6,respectively.Itwasjudgedthat,overall,just under10%ofprojectshave alreadyhad impact to a large extent and thatnearly50%ofprojectshavehadatleastsomeimpact.Theseproportionsincreasedtoca.25%andjustunder80%respectively,withregardtothepotential impactthattheprojects'outcomes may have in the future. These results indicate that bottom‐up researchschemescanleadtosubstantialimpactontheeconomy,societyorpolicy‐making.Itwill,however,besomeyearsbeforetheimpactofERC‐fundedprojectscanbefullyassessed.

Figure5: Impact outside the scientific domain (e.g. on economy, on society, on policy‐making)thatisalreadyapparent.

Figure6: Impact outside the scientific domain (e.g. on economy, on society, on policy‐making)thatcouldmanifestitselfinthefuture.

8

Thepatternthatemerges isconsistentwiththenatureof thetypeofresearch funded.Onewouldexpecttheimpactoffrontierresearchontheeconomyoronsociety,inmostcases,tooccurinthemediumtolongterm,andthisispreciselywhatisshowninFigure5 andFigure6.When considering impact in the futuremuch higher figures are to beexpected. Conclusions

The overall picture that unfolds from this qualitative evaluation of ERC completedprojects seems tobe consistentwith theambition setby theERCScientificCouncil inline with its remit to support high‐risk/high‐gain projects. The main findings thatemergedfromthisevaluationare:

Morethan70%oftheprojectsevaluatedhavemadescientificbreakthroughsofmajoradvances;

About 30% of the projects evaluated have made contributions that can beregardedasincrementalornotsignificant;

In general, the projects have had a very positive impact on the career of theprincipalinvestigators;

Theprojectshavestronglycontributedtotheconsolidationofresearchteams; Alargefractionoftheprojectsevaluatedinvolvedinterdisciplinaryresearch; Closeto50%ofprojectsalreadyhadsomeapparentimpactontheeconomyand

society, and nearly 10% had a major impact to date, which underlines theimportanceoftheERCapproachofgivingresearchersthefreedomtoundertakecuriosity‐driven frontier research. Without having societal impact in mindinitially,thisbottom‐upapproachdeliversinthisrespect.

Itistheexperts’estimatethatatleastthreequartersoftheresearchoutputareexpected to have an impact on the economy or on society in themedium andlongterm.

These findings give, however, only an indication of the first projects funded by ERCshortlyaftertheircompletionandthereforecannotbeextrapolatedtothewholepoolofERC‐fundedprojects,sincemostprojectsareyettobecompleted.

Futureexercisesareplanned tocontinue to follow latercompletedprojects througharandomsampleastherewillbetoomanytoevaluateallofthem.Thenextroundswillincorporateimprovementsbasedonthelessonslearnedfromthisfirstpilotexercise.