bibliometric indicators to measure frontier research · 2017-03-10 · compare the expert-based...
TRANSCRIPT
Bibliometric Indicators to Measure Frontier Research Does the ERC Fund Frontier Research?
AIT:
Edgar Schiebel
Marianne Hörlesberger
Thomas Scherngell
Dirk Holste
CNRS INIST:
Ivana Roche
Dominique Besagni
Claire François
Pascal Cuxac
DBF - Development of a bibliometric model for the identification of frontier research
Coordinated Support Action
support the ongoing monitoring and evaluation work as well as future
strategy and policy development
The ERC peer review system is at the very heart of the ERC's operations
and a crucial element in realising its scientific strategy. Analysis is needed
to monitor the effectiveness and efficiency of the peer review process
(including its implementation) and to understand the particular dynamics and
considerations at play in the ERC Monitoring process of selecting
successful applicants, taking account of the interplay between scientific
and administrative aspects of the process.
2 14.03.2013
DBF - Basics
From 2009-09-01 to 2013-02-28
Partners:
AIT Austrian Institute of Technology
Institute for Scientific and Technical Information (INstitut de l’Information
Scientifique et Technique - INIST) of the French National Center for
Scientific Research (CNRS)
3 14.03.2013
DBF - stages
Design, implement and test a bibliometric approach – based on aspects of
frontier research identified in grant proposals
Compare the expert-based peer review and bibliometric evaluation and draw
lessons learned
Engage with stakeholders and identify aspects of the bibliometric approach
to support the selection of (high-quality, risk-affine and reward-delivering)
frontier research
4 14.03.2013
Content
Basic ERC review process
Model and approach (basic process, attributes for frontier research)
Indicators (definitions, calculation, results and discussion)
Risk
Applicability
Interdisciplinarity
Novelty
Econometric decision model and results (today - afternoon)
5 14.03.2013
Basic ERC Review Process
6 14.03.2013
Proposals for research grants
ERC peer review panels
yes no Frontier research Other research
Model and Approach
Does the ERC Promote Frontier Research in Europe?
7 14.03.2013
Application of bibliometric
indicators for frontier
research on proposals
Do the decision criteria for
„yes“ or „no“ fulfill attributes of
frontier research?
Application of the decision
modell on the outcome of the
ERC peer review panels
Define Frontier Research
Use the HLE-Group Definition for Frontier Research
8 14.03.2013
Proposals for research grants
ERC peer review panels
yes no Other research
Attibutes for frontier research*
•Risk
•Applicability
•Interdisciplinarity
•Novelty
(*) Frontier research. The European Challenge, High Level Expert Group, EC 2005
How to understand attributes of frontier research? with a perspective to establish bibliometric indicators
9 14.03.2013
Frontier research
key attribute (EC, 2005)
Interpretation
Risk: “Frontier research is an intrinsically risky
endeavor…Researchers must be bold and take
risks.…”
Does the investigator take personal risk by
extending his existing research background by a
new research issue?
Applicability (Pasteuresqueness): Frontier
research “… may well be concerned with both
new knowledge … and with generating
potentially useful knowledge at the same time.”
Does the activity of the investigator indicate
applicability of his research?
Science of interdisciplinary nature: Frontier
research “(…) pursues questions irrespective of
established disciplinary boundaries. It may well
involve multi-, inter- or trans-disciplinary
research …”
Does the investigator use research issues,
concepts, methods, etc. from different disciplines?
Novelty: “...stands at the forefront of creating
new knowledge and developing new
understanding...”
Does the proposal use the latest research as it‘s
intellectual base? (Timeliness)
Is the research agenda part of a new upcoming
research front in the field? (Innovativeness:
Similarity to new research issues) EC (2005): Frontier research. The European Challenge, High
Level Expert Group
Risk Indicator
Does the investigator take personal risk by extending his existing
research background by a new research issue?
10 14.03.2013
Proposal Principal investigator
Proposal data:
Reference list
Reference lists of
previous publications:
Risk indicator:
similar: no risk
different: high risk
Risk Indicator: Definition
Key Attribute Risk: “Frontier research is an intrinsically risky
endeavor.…Researchers must be bold and take risks.…”, (EC, 2005)
Formalisation:
Extending the familiar research environment
• means to establish or enter a new or different knowledge base
• Knowledge bases are documented in the cited references of
– the proposal
– the publications of a scientist and
Hypothesis: The less similar the two reference profiles are, the more
independent and risky is the research
11 14.03.2013
Calculation of the RISK indicator
12 14.03.2013
ERC data Research
project proposal
-
External data
Data preperation
Extraction of the cited references
Standardizing the notation of the references
Searching for the PI and his/her references
Extracting the cited references
Cutting the latest work
Calculating the frequencies of the cited references Si
mila
rity
by
cosi
ne
of
refe
ren
ce v
ecto
rs
2006
Results for Risk Indicator: Panel PE7 2009 / StG
13 14.03.2013
Project ID cos
239700 0
239827 0
239987 0
239726 0,0220
240166 0,0223
239986 0,0234
240049 0,0241
240432 0,0263
240475 0,0293
239640 0,0387
240631 0,0393
240236 0,0443
240108 0,0490
239954 0,0583
239720 0,0595
240205 0,0676
240555 0,0717
240241 0,0786
240218 0,0873
240686 0,1032
Sorted by ascending cosine (cos),
the first twenties: more “risky” part
of the distribution
The cosine for the project ID
239700 = 0 indicates that the
references listed in the project and
the references of previous
publications of the PI are disjoint.
The higher the cosine, the closer
to the value 1 the more references
are common in the proposal and in
the recorded paper of the PI.
3 marked proposals are successful
ones
PE7 Systems and communication engineering: electronic, communication, optical and systems engineering
Discussion of the Risk Indicator
Comparison of cited references:
indicates differences in the intellectual base of the proposal against previous publications
does not include qualitative aspects
is just a hint for possible risk: necessary but not sufficient
At present state high workload due to standardization of references time consuming search for previous publications
14 14.03.2013
Applicability (Pasteuresqueness) Indicator
Does the activity of the investigator show a probability for the
application of his research?
15 14.03.2013
Proposal
Indicator: Number of
patents
Self citations in
the reference list Extraction of
Journals
Reference list of
journals for
applied or basic
research
Indicator: share
of applied journals
Pasteuresqueness Indicator: Definition
Key attribute applicability: Frontier research “… may well be concerned with
both new knowledge about the world and with generating potentially useful
knowledge at the same time. Therefore, there is a much closer and more
intimate connection between the resulting science and technology, with few
of the barriers that arise when basic research and applied research are
carried out separately.” (EC, 2005)
Formalisation of the Indicator:
Number of patents
Hypothesis: The patents applied or invented by the PI represent a
general attitude of the PI of whether or not he or she is driven by the aim
to create application relevant results
Share of Journals where the PI publishes categorized as applied vs.
fundamental research
Hypothesis: The category of the journals in which the PI uses to publish
can give an indication of the orientation applied/fundamental of the PI’s
research
16 14.03.2013
Calculation of Pasteuresqueness
17
Project IDPasteuresqueness -
Patents
Pasteuresqueness -
Applied part of PI's
publications
242837 3 1
242783 1 1
243028 1 1
242673 0 1
242859 0 1
242878 0 1
242949 0 1
241260 0 0.9523801
243171 0 0.9473684
243195 2 0.9375
242915 5 0.9090909
243073 3 0.875
242596 0 0.8333333
243113 1 0.7619048
241684 0 0.75
243033 0 0.75
242726 0 0.7368421
242771 0 0.7
242564 0 0.6666667
242699 3 0.5555556
241222 0 0.5333333
242623 0 0.4545455
242820 0 0.2272727
242381 0 0.2
243118 0 0.1538462
240381 0 0.125
242293 0 0.0625
242641 0 0.0625
242754 0 0
242772 0 0
242796 0 0
243024 0 0
243137 0 0
Example: Panel LS9 / Call 2009 / Starting Grant
18
the proposals sorted by
decreasing value of
Pasteuresqueness (journals)
Higher number of patents in the
first half, lower number of patents
in the second half
the 7 successful proposals of 32
are highlighted in green
2 successful with higher
pasteuresqueness
5 successful with lower
pasteuresqueness
Only 2 of 7 successful proposals
with patents
LS9: Applied life sciences and biotechnology: agricultural, animal, fishery, forestry and food sciences; biotechnology, chemical biology,
genetic engineering, synthetic biology, industrial biosciences; environmental biotechnology and remediation
Discussion of the Pasteuresqueness Indicator
Simple calculation, direct and straight forward, conclusive
High workload due to data availability (pdf-files)
Future improvement:
Electronically available data
available list of categorized journals
Additional alternative: PI’s co-authors’ affiliations of previous publications – share of companies, hospitals and applied research organizations
19 14.03.2013
Indicator Interdisciplinarity
Does the investigator use research issues, concepts, methods, etc.
from different disciplines?
20 14.03.2013
Proposals
ERC panels
•PE1
•PE2
•…
•PS1
•…
Assignement
disciplinary tagged keywords by
home panel: „DNA“ - Biology
Extraction of
keywords
Indicator:
number of keywords with
different home panels
Proposals
Indicator Interdisciplinarity: Definition
Key attribute applicability: Frontier research “(…) pursues questions
irrespective of established disciplinary boundaries. It may well involve multi-,
inter- or trans-disciplinary research that brings together researchers from
different disciplinary backgrounds, with different theoretical and conceptual
approaches, techniques, methodologies and instrumentation”, (EC, 2005)
Formalisation of the indicator:
Number of terms from different disciplines
Hypothesis: The more research issues concepts, techniques,
methodologies and instrumentation from different disciplines are
addressed the higher is the interdisciplinary character of the proposal
21 14.03.2013
Interdisciplinarity
22
Example: Panel PE1 / Call 2009 / Starting Grants
23
Proposals of panel PE1 sorted by
decreasing value of addressed
panels
Successful proposals are marked
All 11 successful within the first 12
of 39 proposals
Project ID Share of addressed panels
239885 95
239694 94
239807 94
240518 93
239748 92
239781 92
240123 92
239983 87
239784 85
240127 85
239959 82
239870 81
239902 76
240428 76
240269 73
240074 72
240223 71
240053 70
240471 70
240157 69
239800 68
239814 68
239952 67
240121 67
240693 67
240008 66
240192 66
PE1 Mathematical foundations: all areas of mathematics, pure and applied, plus mathematical foundations of
computer science, mathematical physics and statistics
Discussion of the Interdisciplinarity Indicator
Simple calculation of that indicator, direct and straight forward, theoretically conclusive
Data: all information like abstracts electronically available
Can be calculated for each new call, not sensitive on changing panel definitions or pre-classifications
24 14.03.2013
Indicator Novelty
Does the proposal use the latest research as it‘s intellectual base?
(Timeliness) Is the research agenda part of a research front in a
broader field? (Innovativeness: Similarity to new research issues)
25 14.03.2013
Proposal
Indicator Timeliness:
year of cited
references
Textual
information Research fronts:
PASCAL
Database
Indicator Innovativeness:
similarity of the proposal to
research fronts
Thank you for your attention