A new role for libraries in research assessments

Wouter Gerritsma

Ticer International Summer School 2012

Contents

Introduction

Research assessments

CRIS and repository

Advanced bibliometrics

Alternatives

Why in the library

How can you start?

Introducing Wageningen UR Library

Wageningen University & Research Centre is the results of the merger between Wageningen University and the former DoA research institutes

Life Sciences is our domain

Some 6500 researchers and 10000 students

Strong international focus

Ranked 75th worldwide, 2nd in the Netherlands, 17th in Life Sciences ranking of THE WUR 2001-12 ranking

Bibliometrics at Wageningen UR Library

Since the 1990s' few citation analysis with SciSearch on Dialog and DIMDI

2001: Web of Science

●Collection analysis

●First exercises with bibliometrics

2004: Essential Science Indicators

●Bibliometric analysis for graduate school WIAS

2008: Start of implementation of bibliometric services on our (metadata-)repository Wageningen Yield

National research assessments elsewhere

Australia: The Excellence in Research for Australia (ERA) http://arc.gov.au/era

UK: Research Excellence Framework http://www.ref.ac.uk

●The pilot exercise concluded that citation information is not sufficiently robust to be used formulaically or as a primary indicator of quality in the REF; but there is scope for such data to inform and enhance the process of expert review.

Research assessment in the Netherlands

Supervised by VSNU/QANU

●6 year cycle for external peer reviews

●After 3 year midterm review

●Unit of analysis (in Wageningen): Graduate schools

Citation analyses are not stipulated in the current Standard Evaluation Protocol (SEP).

●This has become mandatory at Wageningen UR, als at the social sciences department and for the research institutes

SEP Criteria

Quality (including international academic reputation and PhD training)

Productivity (the relationship between input and output)

Societal relevance (including valorisation)

Vitality and feasibility (the ability to react adequately to important changes in the environment).

Metis, our CRIS

Metis is a Current Research Information System (CRIS)

Data entry at chair group level

Quality control by the library

●Locating full text (uploading to e-depot)

●Maintenance journal lists

●Document type assignation

Compulsory output registration

●Research assessments only on metis registered pubications

Information on all labour relations of faculty and staff

Information on all projects

Wageningen Yield

Wageningen Yield (WaY) is the bibliographic database of all output of Wageningen UR and OA repository

Comprehensive for the university since 1976, for the research institutes since 1996

Metadata records as well as full text

Ca. 180,000 metadata records

Ca. 37,000 full text OA publications

Since 2003 fed by metis, ca. 90,000 records

Synchronization each night with updates from metis

Klik op het pictogram als u een afbeelding wilt toevoegen

Collecting citation data from Web of Science

WoS record Identifier (UT)

Thomson Reuters Article Match Retrieval service http://wokinfo.com/directlinks/amrfaq/

API

●Input DOI or ISSN/Journal Title/Volume/Start Page

●Return: UT

●Input: UT

●Return: Times Cited

Full screen image with title

Klik op het pictogram als u een afbeelding wilt toevoegen

How are we able to compare numbers?

Scientist Z. Math has a publication from 2001 with 17 citations

Scientist M. Biology has a publication from 2007 with 32 citations

Baselines for Mathematics

Baselines for Molecular Biology

A quantitative example

Bouma, J, Bulte, EH, & DP van Soest (2008) Trust, Trustworthiness and Cooperation: Social Capital and Community Resource Management. J. Env. Ec. & Mngmt 56(2)155-166.

●Cited 12 times

Journal of Environmental Economics and Management from journals menu in ESI:

●Economics & Business

Baseline data for Economics & Business (from ESI)

●Article from 2008: Average: 3.62 Citations; Top 10%: 9 citations; Top 1% 27 citations

RI = 12 / 3.62 = 3.31

Advanced bibliometric indicators

Follow Moed (1995) as closely as possible; but.....

Web of Science is used for citation data

●We can’t make corrections for self citations

Essential Science Indicators for baseline data (World average, Top 10% and Top 1%)

●Limited number of research fields (22)

BUT:

We can determine the representativeness of the citation analysis!

Representativeness

Publication type #Pubs

Refereed articles 324

Non-refereed articles 7

Books 1

Refereed book chapters 36

Non-refereed book chapters 13

PhD Theses 45

Conference papers 137

Total Academic Publications 563

Representativeness

Publication type #Pubs WoS Repr.

Refereed articles 324 288 89%

Non-refereed articles 7

Google Scholar 1

Refereed book chapters 36

Non-refereed book chapters 13

PhD Theses 45

Conference papers 137

Total Academic Publications 563

Slice and dice any way you want

Standard citation reports are created for any set of publications

Accountability is of utmost importance!

Register once use many

Register once use many

Register once use many

Register once use many

Matching Way and WoS

WoS: 9577 articles

WaY: 10933 articles

Missing in Way: 807 articles

Missing from WoS: 1159 articles

1161 peer reviewed articles not in ISI journals

Sources of citation data

Web of Science

Scopus

Google Scholar

Microsoft Academic

SciFinder; Psychinfo

Citeseer and other open access repositories

Various Altmetrics initiatives

Differences in citations reflected in h-index

Differences in citations reflected in h-index

Differences in citations reflected in h-index

Web of Science & Incites

Citation data (includes also citations from other databases on Wok)

API to download citation data

Baselines from ESI

"New" product InCites

Scopus & Strata

Citation data obtainable through an API

Benchmarking with SciVal Strata, no API yet

Not yet fully developed

Google Scholar

Give them a few more years

Coverage?

Ghost citations

Content duplication

Benchmarking?

Altmetrics http://altmetrics.org/manifesto/

Quickly developing

●PLoS article level metrics

●ScienceCard

●Total-Impact

●Readermeter

●Microsoft Academic Search

●etc.

Wouters, P. & R. Costas (2012). Users, narcissism and control. Utrecht, NL: SURFfoundation. http://www.surffoundation.nl/en/publicaties/Pages/Users_narcissism_control.aspx.

Why in the library?

Library is the functional manager of CRIS / repository because of wide experience with bibliographic metadata

Library manages contracts with publisher(s) of external databases that are being used

Library has experience in developing and maintaining large databases

Library has ample experience in searching complicated databases such as Web of Science

Advantage of CRIS/repository combination

Improvements in publication lists, etc. recorded

Knowledge of, and experience with bibliometric analyses is better institutionalized

Clarity / transparency for researchers

Analysis of a single unit of the institute offers advantages for whole institute

Better understanding of our own researchers

●We know where they publish

●We know what they cite

●We know something about their impact

Raising library awareness

Improvement of the (metadata) quality in the repository

Quality has lead to compulsory registration for research assessments

Presentations for research groups during the preparation for peer reviews

Presentations based on detailed studies of single groups

Library gives advice on elements for publication strategies for groups and individuals

●there is a huge demand for these workshops

Closing the circle

With the coupling of publications with WoS

We have gained insight in the relation

●Research group – Researchers – Publications – Reference list

●It is feasible to assign journal usage at faculty level, or more detailed (chair groups)

Journal title Total AFSG ASG ESG Imares PSG SSG

NATURE 2584 511 341 753 93 989 59

PNAS 2467 787 325 166 20 1225 29

SCIENCE 2303 529 239 594 52 970 99

APPLIED AND ENVIRONMENTAL MICROBIOLOGY 2257 1320 257 139 12 696 27

PLANT PHYSIOLOGY 1597 379 4 58 0 1296 2

JOURNAL OF BIOLOGICAL CHEMISTRY 1543 931 223 13 6 379 8

But where should you start?

Courses on bibliometrics

●CWTS http://www.socialsciences.leiden.edu/cwts/

●ESSS http://www.scientometrics-school.eu/

Conferences

●ISSI http://www.issi-society.info/aboutconf.html

●STI http://2012.sticonference.org/

Journals

●Scientometrics

●Journal of Informetrics

Books

●Moed, H.F. (2005). Citation analysis in research evaluation. Dordrecht, Springer.

Thank you!

On the Web:@wowterwowter.netwww.slideshare.net/wowter

http://tinyurl.com/7r67fmm