Citation Impact: An Introduction to Bibliometrics for Researchers

Breeda Herlihy (Institutional Repository Manager) b.herlihy@ucc.ie

Ronan Madden (Arts & Humanities Librarian) r.madden@ucc.ie

02/05/2023 UCC Library Introduction to Bibliometrics

Aims and Outcomes

Aim:• To look at methods of identifying and

interpreting research performance data as a measure of research impact

Outcomes:• Use citation analysis tools (as provided by

Web of Science, Scopus, Google Scholar) to evaluate research impact

• Understand the limitations of bibliometrics• Utilise publishing strategies to improve

citation performance02/05/2023 UCC Library Introduction to Bibliometrics

Format

1. IntroductionResearch evaluation, peer-review,

citation analysis & bibliometrics2. Citation Impact

Citations per paper, individual, institution, country3. Journal Impact

Impact factor, SJR, etc.4. Caveats5. Publishing Strategies incl. Open Access

02/05/2023 UCC Library Introduction to Bibliometrics

‘Scholarly’ Communication• Traditional Approach: From informal to

formal, communication of research at every stage; evaluation and feedback from peers at every stage

1. Initial/Informal: meetings, discussions, seminars, emails, funding applications, blogs, social networking, twitter2. Report on-going research at a conference3. Publication in an academic journal, scholarly monograph4. Indexed in research/bibliographic databases, catalogues, repositories

02/05/2023 UCC Library Introduction to Bibliometrics

Evaluating Research

• Qualitative approach: ‘Peer-reviewed’Possible at every stage of research

process from funding to publication. Types: Double-blind, Single-blind, Open etc.

Limitations: See ‘Peer-review: a guide for Researchers’

• Quantitative approach: Bibliometrics/Citation analysis– Measures ‘research impact’ (not

‘quality’)02/05/2023 UCC Library Introduction to Bibliometrics

Research Impact

• Not just bibliometrics (which focuses on journals)Other methods of dissemination: monographs, book chapters, conferences, reports, digital objectsAlso includes:- research income generated- funding & awards- number of PhD students/graduates- patents and licences obtained- impact on society, ‘cultural capital’ (social, economic, cultural)02/05/2023 UCC Library Introduction to

Bibliometrics

Citation Impact/bibliometrics

‘ The branch of library science concerned with the application of mathematical and statistical analysis to bibliography; the statistical analysis of books, articles, or other publications’ Oxford English Dictionary

• Quantitative measures…• Impact of a paper, a journal, an individual

researcher, institution/university, country etc.

• Controversial? – each should be judged on their own merits within context. Fair? Objective? Cost effective?

• Caveats – more later…02/05/2023 UCC Library Introduction to

Bibliometrics

Who’s interested in bibliometrics?

• Governments and funding agencies- return on investment, grant applications [See video]

• Institutions/ Universities- benchmarking, strengths/weaknesses, recruiting, return on investment

• Units within universities e.g. schools, colleges

- benchmarking, strengths/weaknesses, collaboration, recruiting [video]

• Individual researchers- promotion/recruitment, tenure, grants [video]

• Librarians! – subscriptions, value for money02/05/2023 UCC Library Introduction to

Bibliometrics

Citation Analysis (Bibliometrics)

• The number of times a paper, researcher, journal is cited by others; assumes that influential authors and important works are cited more often than others

• 2 elements: - 1. Citation Impact: measures the

influence of particular papers (times cited) and authors (e.g. H-index). Data can be used to help evaluate departments, universities, countries

- 2. Journal Impact : measures the influence that particular journals have within a field of study

02/05/2023 UCC Library Introduction to Bibliometrics

Where to find the data:1. Web of Knowledge (Thomson Reuters) incl. Web of Science’s 12,300+ peer-reviewed journals, includes ‘Journal Citation Reports’ and ‘Essential Science Indicators’. Data is used by ‘InCites’. See title listing.

2. Scopus (Elsevier) 19,800+ peer-reviewed journals; 1996 on. Scimago Country & Journal Rank Database details journals and country-specific scientific indicators developed from the information contained in Scopus

3. Google Scholar provides links to ‘cited by’ information, and some metrics. Data is harvested by ‘Publish or Perish’ software to produce more detailed metrics

4. ‘Altmetrics’ & Web metrics is an emerging area02/05/2023 UCC Library Introduction to

Bibliometrics

SCOPUS & WOS coverage comparison

Scopus: from 1996 only i.e. width rather than depth

11,377 8,432934

Source: JISC http://adat.crl.edu

02/05/2023 UCC Library Introduction to Bibliometrics

Source: http://www.ndlr.ie/myri/

02/05/2023 UCC Library Introduction to Bibliometrics

Citation Impact

The impact of a paper or group of papers i.e. number of times an article/paper is cited by others.Can be used to measure impact of a researcher, institution, country etc.

- Web of Science (part of Web of Knowledge) ‘Times cited’, also includes ‘Cited Reference Searching’ to find articles that cite a person’s work.- Scopus see ‘Cited by’- Google Scholar: provides ‘Cited by’ information

02/05/2023 UCC Library Introduction to Bibliometrics

Example: Number of times citedArticle Title: Identification of pancreatic cancer stem cellsAuthor(s): Li CW, Heidt DG, Dalerba P, et al.Source: CANCER RESEARCH   67 (3): 1030-1037   Published: FEB 1 2007

1. Go to Web of Knowledge2. Pick ‘Web of Science’3. Enter the Title to locate article4. Note Number of ‘Times Cited’5. Go to Google Scholar6. Type article title in box7. Note ‘Cited by -’8. Go to Scopus9. Enter title to locate article10 Note number ‘Citations’

02/05/2023 UCC Library Introduction to Bibliometrics

Example: Number of times citedArticle Title: Identity, politics and conflict in dockland

development in Cork, Ireland: European Capital of Culture 2005

Author(s): O’Callaghan, C. and Linehan, D. Source: Cities 24(4): 311-323   Published: 2007

1. Go to Web of Knowledge2. Pick ‘Web of Science’3. Enter the Title to locate article4. Note Number of ‘Times Cited’5. Go to Google Scholar6. Type article title in box7. Note ‘Cited by -’8. Go to Scopus9. Enter title to locate article10 Note number ‘Citations’

02/05/2023 UCC Library Introduction to Bibliometrics

Exercise:

Mullally B.J. et al. (2008) ‘Prevalence of smoking among bar workers prior to the Republic of Ireland smokefree workplace legislation’, Irish Journal of Medical Science, 177(4): 309-16

• How many times has this article been cited in Web of Science, Scopus and Google Scholar?

02/05/2023 UCC Library Introduction to Bibliometrics

Citation Impact expanded

• Individual Researchers (promotions, funding etc.):

- Cited reference search: Web of Science- H-index: -highest number of papers a researcher has that have each received at least that number of citations e.g. someone with h-index 10 has written 10 papers that have each had at least 10 citations.

Where to find it (3 different versions):- Web of Science: ‘Author Finder’ and ‘Create Citation Report’.- Scopus: ‘Author Search’ and ‘Citation overview’ N.B. 1996 on- Publish or Perish and Google Scholar (Humanities?)

[In each case: difficult to identify all of a researcher’s work]

02/05/2023 UCC Library Introduction to Bibliometrics

Example:• Find a H-index for a researcher Heidt,

D.G. working in the field of Biomedicine

Use: • Web of Science• Scopus• Google Scholar ‘My citations’• Publish or Perish http://www.harzing.com/pop.htm

N.B. Heidt D v Heidt DG (ResearcherID can help)

02/05/2023 UCC Library Introduction to Bibliometrics

Citation Impact expanded

• An Department/School, Institution, Region, Country:

o Web of Science- ‘Organization Enhanced’ search:

‘University College Cork’ Select from the indexo Essential Science Indicators (WOK):

NATL UNIV IRELAND UNIV COLL CORKo Scopus – Affiliation search o Scimago Journal & Country Ranko InCites: see here, institutional

performance, subject areas, comparison, collaboration

N.B. conference papers and monographs are under-represented

02/05/2023 UCC Library Introduction to Bibliometrics

University rankings

• Shanghai Rankings: uses Web of Science• Times Higher Education uses Web of

Science (normalised citation impact 32.5%)

• QS Rankings uses Scopus (switched in 2007) 20%

• HEEACT: uses Web of Science• Leiden Ranking WoS data only• U-Multirank: a new European ranking

initiative • Other metrics: WoS docs per academic,

relative citation impact etc.02/05/2023 UCC Library Introduction to

Bibliometrics

Journal Impact

The average number of citations to papers within a particular journal (journal performance metrics)

Why useful?- Deciding where to publish to maximise impact- Librarians in managing

acquisitions/subscriptions

Where are the metrics?2 main sources:- Web of Knowledge: ‘Journal Citation Reports’- Scopus and Scimagosee also ‘Publish or Perish’

02/05/2023 UCC Library Introduction to Bibliometrics

Journal Impact Factor

‘Journal Citation Reports’ (Journal Performance Metrics)

Part Thomson Reuter’s Web of Knowledge (the original)

• N.B. Journal Impact Factor: average number of times articles from the journal published in past two years have been cited in JCR year

• Journal Immediacy Index: average number of times article is cited in the year it is published

• Five year Impact Factor: average number of times articles from the journal published in past five years have been cited in JCR year

02/05/2023 UCC Library Introduction to Bibliometrics

Journal Impact Scopus and Scimago

Scimago Country & Journal Rank Database: (free) journals and country scientific indicators developed from information in ‘Scopus’

• N.B. SJR: ‘Scimago Journal Rank’ reflects prestige of source where such citations come from. Covers a 3 year citation window.

• SNIP: ‘Source normalized impact per paper’: corrects for differences in the frequency of citation across research fields- citations given a weighting based on the total number of citations in that field. N.B. Scopus only

02/05/2023 UCC Library Introduction to Bibliometrics

Journal Impact Other metrics

• Google Scholar Journal Metrics

• Eigenfactor : (free on the web) citations from highly-ranked journals weighted to make a larger contribution to the eigenfactor than those from poorly-ranked journals (similar to SJR)

• Publish or Perish: journal h-index- the number of a journal’s papers that have been cited at least ‘h’ times e.g. h-factor 10 means that 10 papers have been cited at least 10 times

02/05/2023 UCC Library Introduction to Bibliometrics

Example: Impact Factor

1. What is the impact factor of the Journal ‘The Lancet’?1. Go to ISI Web of Knowledge2. Click on ‘Select a database’3. Pick ‘Journal Citation Reports’

4. Choose ‘JCR Science Edition’ and ‘Search for Specific Journal’5. Click on ‘Submit’6. Type ‘Lancet’ in box and search for ‘Full Journal Title’

2. What are the highest impact journals for Oncology?1. Go to ISI Web of Knowledge2. Click on ‘Select a database’3. Pick ‘Journal Citation Reports’4. Choose ‘JCR Science Edition’ and ‘View a Group of Journals by Subject Category’6. Click on ‘Submit’7. Scroll down to ‘Oncology’8. Click on ‘Submit’9. Sort by Impact factor

02/05/2023 UCC Library Introduction to Bibliometrics

Exercise• What is the 2011 impact factor for the

journal ‘Applied Linguistics’?

• What is the 2011 impact factor for the journal ‘Cell’?

• What Sociology journal had the highest impact factor for 2011?

• What Psychology journal had the highest impact factor for 2011?

02/05/2023 UCC Library Introduction to Bibliometrics

Journal Impact Other Sources

• Journal Quality List (Harzing) (mainly business-related subjects)

• ERA: Excellence in Research for Australia produce a list of ‘eligible publications’

• Other national systems: e.g. New Zealand, Norway, Poland, Denmark etc.

• ERIH: European Reference Index for the Humanities European humanities research

• Also : various subject specific lists available

02/05/2023 UCC Library Introduction to Bibliometrics

Other Metrics• Microsoft Academic Search• Altmetrics: attempt to track the impact of

scholarly works in the social web e.g. iopscience and Plos

• Books? Library holdings analysis: OCLC Worldcat ‘Libraries Worldwide’, Book sales, Book citation index. Also : who published it, peer-review process

• Data citation index• Webometrics: quantifying the impact of web

sites e.g. http://repositories.webometrics.info/

http://www.webometrics.info/02/05/2023 UCC Library Introduction to Bibliometrics

How many times cited?

• Wakefield, A. J.; Murch, S. H.; Anthony, A. (1998) ‘Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children’. Lancet, 351(9103): pp 637-41

Note: Citation data is then used to create h-index for authors, impact factor of the journal, metrics for the university etc. etc.02/05/2023 UCC Library Introduction to

Bibliometrics

How many times cited?

Stapel, DA. And Koomen, W (2001).“I, we, and the effects of others on me: How self-construal level moderates social comparison effects.” Journal of Personality and Social Psychology 80(5): 766-781.

Note: Citation data is then used to create h-index for authors, impact factor of the journal, metrics for the university etc. etc.02/05/2023 UCC Library Introduction to

Bibliometrics

Bibliometrics- limitationsUseful tools, but not a measure of true research

quality e.g.:

• current popularity of topic & availability of article/journal: increases citations / higher IF

• IF (and H-Index) based on average over all articles: underestimates citations of top cited articles, IF exaggerates number of citations of the average article

• comparison of impact factors and H-Index across different fields is invalid e.g. not as relevant for literature (books citing other books) . 80% of humanities citations are not included in WoS.

• What about books, book chapters, conference papers, digital humanities etc.?

02/05/2023 UCC Library Introduction to Bibliometrics

Bibliometrics- limitations (contd.)

• Coverage differs in each resource (Web of Science, Scopus, Google Scholar); none is complete and reliable e.g. for IF only WoS database journals used; undercounts number of citations from other journals e.g. ‘less-developed’ countries, other languages, national & niche journals etc.

• Why have the papers been cited? e.g. Review articles, Wakefield

• 90% of papers are never cited, does this mean they are all if inferior quality?

• Takes time for papers to collect citations, especially in some fields

• Culture of systematic review and comprehensive citing not established in every field of studies

• Question of self-cites, cronyism (reciprocal citing), IF manipulation

02/05/2023 UCC Library Introduction to Bibliometrics

Bibliometrics- limitations (contd.)

• H-index is a poor metric for young researchers• H-index underestimates the importance of

seminal articles• Pressure on academics to publish may lead to a

drop in quality, ‘over publication’, implications for peer-review system?

• Google metrics are not yet reliable enough in isolation (what dataset is used? lack of quality control: citations from unpublished sources, web sites, early versions of articles etc.)

• Pressure on Library budgets – subscribe to all the ‘top’ journals

• Citation patterns are becoming increasingly dispersed, lessening the importance of the impact factors

02/05/2023 UCC Library Introduction to Bibliometrics

Impact Factor- limitation

• ‘Like nuclear energy, the impact factor is a mixed blessing.’

• ‘The use of journal impacts in evaluating individuals has its inherent dangers. In an ideal world, evaluators would read each article and make personal judgments.’

Garfield, E. (2006) ‘The History and Meaning of the Impact Factor’. JAMA, 295(1): pp 90-93 [Garfield invented scholarly citation in the 1960’s]

02/05/2023 UCC Library Introduction to Bibliometrics

H-index - limitations

‘Obviously a single number can never give more than a rough approximation to an individual’s multifaceted profile, and many other factors should be considered in combination in evaluating an individual.’

Hirsch, J.E. (2005) ‘An index to quantify an individual's scientific research output’. arXiv:physics/0508025 v5 29 Sep 2006(Hirsch proposed the H-index in 2005)

02/05/2023 UCC Library Introduction to Bibliometrics

Remember:• No substitute for human judgment (be

it a paper, researcher or unit that is being evaluated) assess each on its own merits

• Metrics must be used appropriately • Don’t apply in all cases • Must be considered in combination

with other indicators

But: when used appropriately metrics can assist and supplement peer review, while awareness of metrics can assist researchers seeking to increase research impact02/05/2023 UCC Library Introduction to

Bibliometrics

Publishing strategies to improve citation performance

• Good quality research …but it’s not the only driver of citations

• Citation patterns– Studies observing patterns…

statistics• Citation motivations

– Why do researchers cite papers?

02/05/2023 UCC Library Introduction to Bibliometrics

Journal Selection• Journals with high impact factors

• Journals indexed by Web of Science and/or Scopus

• Less specialized journals

• English language journals

• Review / special issue publishing

Article content• Titles, abstracts, keywords

- 10 times more likely to be read than article itself- Indexing databases- Page ranking algorithms- Long titles

• References– Number of references– Highly cited papers

• Co-authors– Number of co-authors– Highly cited co-authors

• Quality– Avoid minimum publishable unit

Your name• Use a middle initial if possible. Be

consistent with the use of variants of your name – e.g. Smith, John A. / Smith J. A. and not Smith,

John /Smith J.• Do you have a publishing doppelgänger?

– Check your name against authors in Google Scholar, Pub Med, Web of Science, Scopus

• Be pedantic about your name and affiliation on publications– Published information sent to databases…GIGO

Your name…a solution on the horizon

• Open Researcher & Contributor ID– International collaboration between

Funders, Research Institutions and Publishers to solve author ambiguity issues

– Unique author identity registry– Data exchange possible between

ResearcherID, LinkedIn and Scopus Author Profiles

– No affiliation necessary and no cost to author

– ORCID integration planned for next version of IRIS

02/05/2023 UCC Library Introduction to Bibliometrics

Your public research identity

• Give full and proper references to your work– Make it easy to find – Make it accessible i.e. link to full text in

CORA• Claim your publications > accurate

author identification– IRIS – Web of Science through Researcher ID– Scopus– Google Scholar through My Citations

http://www.researcherid.com/rid/G-1593-2012

http://0-www.scopus.com.library.ucc.ie/home.url

http://scholar.google.com/

Availability of your publications

• Is your work or a sample of your work available online? ‘Make it frictionless’ “The diffusion of research — publication of working papers, as well as conference and workshop presentations — has a first-order positive impact on the citation rate.”

• If work is not accessible, will it be cited? • Is your work behind a pay wall? • Archive your publications in an open

access repository e.g. CORA, UCC institutional repository

Archive your work in CORA

• Open Access Citation advantage– Archive early and often leads to increased citations to

your workEngineering

Clinical Medicine

Social Sciences

Gargouri & Harnad, 2010: http://eprints.ecs.soton.ac.uk/18493/

Open Access Citation Advantage

Size of OA citation advantage when found (and where explicitly stated by discipline)

% increase in citations with Open Access

Physics/astronomy 170 to 580Mathematics 35 to 91Biology -5 to 36Electrical engineering 51Computer science 157Political science 86Philosophy 45Medicine 300 to 450Communications studies (IT) 200Agricultural sciences 200 to 600

Swan, Alma (2010) The Open Access citation advantage: Studies and results to date. http://eprints.soton.ac.uk/268516/ A summary of studies 27 found an Open Access citation advantage, while 4 reported no OA citation advantage

Promote your publications

• Use social media such as blogs, twitter and Facebook to highlight your research– Develop your online network– A distributed online identity

• Liaise with UCC Research Support Services & Office of Media and Public Relations to arrange the media promotion of your work

• Monitor impact– Monitor impact > Altmetrics Real Time >

Citation alerts longer term– Can tweets prediction citations?

Monitor impact of social media

02/05/2023 UCC Library Introduction to Bibliometrics

Altmetrics• Alternative metrics / Article level

metrics– Download statistics, social media

mentions, mainstream media mentions– Timeliness– Evidence of reach beyond academia– Measures for a diversity of research

outputs • Tools including

– Altmetric , ImpactStory, Plum Analytics

02/05/2023 UCC Library Introduction to Bibliometrics

02/05/2023 UCC Library Introduction to Bibliometrics

Altmetrics

02/05/2023 UCC Library Introduction to Bibliometrics

Altmetric bookmarklet for Chrome, Firefox and Safarihttp://www.altmetric.com/bookmarklet.php

Earlier Web usage statistics as predictors of later citation impact

Brody, T., Harnad, S. and Carr, L. (2006), Earlier Web usage statistics as predictors of later citation impact. J. Am. Soc. Inf. Sci., 57: 1060–1072. doi: 10.1002/asi.20373

Affiliation• Ensuring that the ‘University

College Cork’ identifier is used in all research publications– ensures credit for your publications

and citations is accumulated by UCC

• National Citation Report project– UCC was found to have more than

1,600+ name variants in Web of Science

Aggregation of Citations per Publication across all 22 Fields Web of Science 2002 –

2012 (February 2013)

2002-2006 2003-2007 2004-2008 2005-2009 2006-2010 2007-2011 2008-2012 0

1

2

3

4

5

6

7

8

9

10

Prepared by Deirdre Kearney, Research Support Services, UCC

• Slides:

• Further reading

• Database Tutorials: – Web of Science– Journal Citation Reports– Scopus Tutorials

• MyRi: Measuring your Research Impacthttp://www.ndlr.ie/myri/

02/05/2023 UCC Library Introduction to Bibliometrics

Appendix 1: Criteria for Journal Inclusion

Web Of Science: see http://ip-science.thomsonreuters.com/mjl/

Scopus:

Appendix 2: Additional tools for Computer Science

02/05/2023 UCC Library Introduction to Bibliometrics

Appendix 3: Further metrics for individuals• Egghe's g-index• Zhang's e-index• Contemporary h-index• Age-weighted citation rate (AWCR) and AW-index• Individual h-index (original)• Individual h-index (PoP variation)• Multi-authored h-index

• See: http://www.harzing.com/pop.htm#metrics

02/05/2023 UCC Library Introduction to Bibliometrics

Appendix 4:

• REF: Assessment framework and guidance on submissions

See in particular REF Section 131 ‘Citation data’

02/05/2023 UCC Library Introduction to Bibliometrics