1

Using metrics to your advantage

Fei Yu and Martin Cvelbar

2

Metrics

• Things that can be counted

• number of publications

• citation counts

• Also:

• Journal quality measures – e.g. Journal impact factor

• H-index

• Collaboration metrics

• Metrics: must always be considered in context – significant disciplinary differences, and differences over time.

Using metrics to your advantage

3

Why use publication metrics?

1. Metrics provide supporting evidence of claims individuals/research groups make

• Track record of individuals, groups of researchers

• Contribution to the discipline and/or significance of the contribution

• Evidence of international profile

• Evidence of capacity to collaborate effectively

2. The current environment …

• research evaluation now part of the landscape (ERA) – grant application reviewers, potential employers etc. may expect to see metrics

Using metrics to your advantage

4

Research Quality

• Determined by peers

• peer review

AND/OR

• Inferred by “numbers” – bibliometrics indicators

• Citation counts etc.

Using metrics to your advantage

5

qual·i·ty/ˈkwälətē/

Noun

• the standard of something as measured against other things of a similar kind

• the degree of excellence of something

Given the numbers some meaning!

Using metrics to your advantage

6

Using metrics to your advantage

For papers published in journals within the ESI field of Mathematics in 2002 the average citation rate in WoS is 4.91

In Molecular Biology and Genetics it is 37.24!

7

Citation counts

As a research quality or performance measure the assumption is:

high citations = high quality

BUT

• “…not only the content of scientific work, but also other, in part non-scientific, factors play a role in citing behaviour. Citations can therefore be viewed as a complex, multidimensional…phenomenon”

Lutz Bornmann, Hans-Dieter Daniel, (2008) "What do citation counts measure? A review of studies on citing behavior", Journal of Documentation, Vol. 64 Iss: 1, pp.45 - 80

Using metrics to your advantage

8

Journal Impact Factor

…..

Flawed for a number of reasons…but it may be a useful tool to profile researcher output … …

Using metrics to your advantage

9

Example

“My research output is of very high quality, with publications consistently in good quality journals … “

• What % of your journal articles are in the top 10% by Impact Factor for the Journal Citation Reports categories you publish in?

Using metrics to your advantage

10

Metrics: the standard 5

1. Number of publications

2. Career citation count

3. Citations per paper

4. % cited (or % uncited)

5. h index

These numbers tell us something … but there’s a lot missing … …

Using metrics to your advantage

11

How to use metrics to paint a complete picture

Using metrics to your advantage

12

Provide more detail … …

1. Number of publications

• Profile by document type, subject area, year, journal etc.

2. Number of citations (career citation count)

• Provide an indication of coverage by citation data provider

• Are all outputs included or just articles and reviews?

3. Citations per paper

• I have 35 refereed journal articles, of which 33 are indexed by Web of Science. These articles have received 230 citations, giving an average citation per (indexed) paper of 7 (source: WoS, 05/01/15).

4. Percentage not cited

• Of my 33 indexed journal articles, 3 articles have not been cited by others (9% not cited), and these were all published in 2014 (WoS, 05/01/15).

Using metrics to your advantage

13

And some additional metrics… …

• Citations for an individual paper or sub-set o publications: supporting evidence for outstanding contribution in a particular area.

• Citations per paper by year of publiation and by journal/subject area/FOR code

• Can then benchmark

• E.g. Incites, or use Thomason Reuters Essential Science Indicators for benchmarking

• Or create your own benchmarks using refine reature within WoS

Using metrics to your advantage

14

5. h-index

… ….

Productivity + Impact = Influence

Using metrics to your advantage

15

Does it mean anything?

• Use the h-index measure with care. Citation patters vary across disciplines.

• E.g. h-indices in Medicine are much higher than in Mathematics for example.

• Researchers in different disciplines can not be compared using the h-index.

• Even within the same discipline the h-index should not be used alone as a measure of research quality.

• Researchers in the same discipline at different stages of their careers can not be compared using the h-index.

Using metrics to your advantage

16

Consider the following 3 researchers who have all published 10 papers and all have an h-index of 5

Using metrics to your advantage

Researcher 1 cites per paper

Researcher 2 cites per paper

Researcher 3 cites per paper

Paper 1 100 50 15

Paper 2 90 50 15

Paper 3 70 45 14

Paper 4 50 45 14

Paper 5 35 5 12

Paper 6 4 4 4

Paper 7 0 2 4

Paper 8 0 0 4

Paper 9 0 0 4

Paper 10 0 0 4

Average cites per paper 35.3 20.1 9.0

17

Put it in context … …

• My h-index based on these indexed papers is 10. I have 4 papers (A, B, C, D) with more than 20 citations and 1 paper (E) with 29 citations (source: WoS, 05/01/15). I also have an additional 3 papers not indexed by WoS, with 29 citations based on Scopus data (05/01/15).

Using metrics to your advantage

18

More examples … …

“My work is multi-disciplinary, spanning biochemistry, biophysics and oncology …”

Analyse your WoS articles by WoS subject category to see if this is evidenced in your research output.

“34% of my journal articles are in the WoS Subject Area of Biochemistry and Molecular Biology, with 29% in Biophysics and 16% in Oncology (WoS Subject Areas, 05/01/15)”.

Using metrics to your advantage

19

More examples …

“My research has found (widespread) application beyond my discipline … “

Look at the articles which cite your work in WoS and analyse them by WoS Subject Area.

“Whilst 65% of my articles are published in Physiology (WoS Subject Area), 55% of my citations are from journals with a Subject Area of Cell Biology (source: WoS, 05/01/15).”

Using metrics to your advantage

20

For Early Career Researchers… with a short track record

• Look in detail at the quality of the citations your work is receiving

• What are the impact factors of the journals in which your work is cited

• What fields are the journals in?

• Who is citing your work?

• Who are they?

• Where are they from?

• Why are they citing your work?

• How quickly is your work being cited?

Using metrics to your advantage

21

Beyond bibliometrics…..

Using metrics to your advantage

Altmetrics is the creation and study of new metrics based on the Social Web for analyzing and informing scholarship.

From http://altmetrics.org/,aofesto/

22

Using metrics to your advantage

23

Consider other measures of research impact … … for example:

• Mentions in legislation, government policy etc.

• Academic library holdings

• Look at these “56 indicators of impact”. http://cas-csid.cas.unt.edu/?p=4475

• See also:

Using metrics to your advantage

24

Some questions to think about:

• Have you viewed your download statistics or abstract views in eSpace?  [countries & dates viewed]

• How many of your books are held in libraries worldwide?  [library can provide data]

• Have you been invited to speak at conferences, and were the papers peer-reviewed?

• Has any of your work been commissioned?

• Has your research influenced government or education policy?

• Has there been any measurable social benefit from your research?

• Have you collaborated with others domestically or internationally?

• Have you received any awards?

Using metrics to your advantage

25

Some advice … …

• Use author disambiguation tools

• Google Scholar Citation profile

• ResearcherID

• Scopus authorID

• ORCiD (Open Researcher and Contributor ID)

Using metrics to your advantage