The role of bibliometrics in research management ...

It’s complicated !

Dr. Thed van LeeuwenCenter for Science & Technology Studies (CWTS)

Erasmus Seminar “Research Impact & Relevance”, June 17th 2014

2

Outline

• Bibliometrics and research management context

• Infamous bibliometric indicators

• Scientific integrity and misconduct

• Applicability of bibliometrics

• Take-home messages …

3

Bibliometrics and the research management context

What is bibliometrics ?

• Bibliometrics can be defined as the quantitative analysis of science and technology (development), and the study of cognitive and organizational structures in science and technology.

• Basic for these analyses is the scientific communication between scientists through (mainly) journal publications.

• Key concepts in bibliometrics are output and impact, as measured through publications and citations.

• Important starting point in bibliometrics: scientists express, through citations in their scientific publications, a certain degree of influence of others on their own work.

• By large scale quantification, citations indicate (inter)national influence or (inter)national visibility of scientific activity, but should not be interpreted as synonym for ‘quality’.

‘Classical’ image of the Credibility cycle

Credibility cycle (adapted from Latour and Woolgar (1979) & Rip (1990)

PEER REVIEW

Rise of performance indicators & bibliometrics

• Need for formalised measures• ‘Push’ from science policy (from 1970s onwards)

• Independent of peer review

• New Public Management / Neo-liberalism (from 1980s onwards)

• Matrix-structure science (Whitley)

• Researchers part of international community

Peer review

• But also part of local institutions

Specific management practices (appraisals, external evaluations)

• Institute managers not always part of international expert community

• Tighter forms of management (from the 1990s onwards)

Distance

Extended credibility cycle

8

Infamous bibliometric indicators:JIF & H-index

Definitions of Journal Impact Factor & Hirsch Index

• Definition of JIF:

– The mean citation score of a journal, determined by dividing all

citations in year T by all citable documents in years T-1 and T-2.

• Definition of h-index:

– The ‘impact’ of a researcher, determined by the number of received

citations of an oeuvre, sorted by descending order, where the number

of received citations on that single paper equals the rank position.

Problems with JIF• Methodological issues

– Was/is calculated erroneously (Moed & van Leeuwen, 1996)

– Not field normalized

– Not document type normalized

– Underlying citation distributions are highly skewed (Seglen, 1994)

• Conceptual/general issues

– Inflation (van Leeuwen & Moed, 2002)

– Availability promotes journal publishing

– Is based on expected values only

– Stimulates one-indicator thinking

– Ignores other scholarly virtues

Deconstructing the myth of the JIF…• Take the Dutch output

• Similar journal impact classes• Focus on publications that belong to the top 10% of their field

0,0%

5,0%

10,0%

15,0%

20,0%

25,0%

30,0%

35,0%

40,0%

A (0 >MNJS <=0.40)

B (0.40 > MNJS <= 0.80)

C (0.80 > MNJS <= 1.20)

D (1.20 > MNJS <=1.60)

E (MNJS > 1.60)

Problems with H-index• Bibliometric-mathematical issues

– mathematically inconsistent (Waltman & van Eck, 2012)

– conservative

– Not field normalized (van Leeuwen, 2008)

• Bibliometric-methodological issues– How to define an author?

– In which bibliographic/metric environment?

• Conceptual/general issues– Favors age, experience, and high productivity (Costas & Bordons, 2006)

– No relationship with research quality

– Ignores other elements of scholarly activity

– Promotes one-indicator thinking

The problem of fields and h-index …• Spinoza candidates, across all domains …

• Use output, normalized impact, and h-index

Soc

HumMat

Soc

Eng

Psy

Eng ChePsyMed

Med

Che

Med

Med

Phy

PhyBio

BioPhy

Psy

Env

Phy

Med

Bio

MedMed

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

0 50 100 150 200 250

TOTAL PUBLICATIONS

CPP

/FC

Sm

Med

Med

Bio

MedPhy Env

PsyPhy

BioBioPhy

Phy MedMed

CheMedMed PsyChe

EngPsyEngSocMatHum

Soc

0

10

20

30

40

50

60

0 50 100 150 200 250

TOTAL PUBLICATIONS

H-in

dex

14

Scientific integrity and misconduct

Scientific integrity and misconduct

• An often mentioned cause for any kind of misconduct is the “Publish or Perish” culture.

• As such, that is supposed to force individuals to produce more and more output, ….

• … as these are considered the building blocks of an academic career !

• However, pressure is on all of us !

Effects of the “Publish or perish” culture: the case of J.H. Schön

0

200

400

600

800

1000

1200

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

JH Schon

Effects of the “Publish or perish” culture: the case of D. Stapel

0

50

100

150

200

250

300

350

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Impact Stapel DA

D. Stapel caused the Netherlands to be the country ranked first in the field of Social psychology, when it comes to retractions !

18

Applicability of bibliometrics

Output of a university …

Output, and the way it is represented in Web of Science!

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

(Bio)medicine

Economics & Management

Humanities

Law

Social sciences

All Publications

WoS Publications

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

(Bio)medicine

Economics & management

Humanities

Law

Social sciences

BOOK

CASE

CHAP

CONF

GEN

JOUR

MGZN

PAT

RPRT

THES

Output, and the wide variety not covered in Web of Science !

20

Take home messages

Take-home messages on journals

• Journals tend to publish positive/confirming results.• Editorial boards are driven by market shares as well !• Therefore, selection is harsh, and rejection rates are high

Take-home messages on data• Data are not frequently published.• Therefore, they do not give any credits for the producers!• This keeps most work non transparent• Database with negative results is necessary

Take-home messages on bibliometrics• Ask yourself the question “What do I want to measure ? ”• And also “Can that be measured ? “ • Take care of proper data collection procedures.• Then, always use actual and expected citation data.• Apply various normalization procedures (field, document, age)

• Always have a variety of indicators.• Always include various elements of scholarly activity.

• And perhaps most important, include peer review in your assessment procedures !!!

23

Thank you for your attention!

Any questions?Ask me, or mail me

leeuwen@cwts.nl

24

Appendix slides

25

Development of authorship across all domains of scholarly activity

1981

1983

1985

1987

1989

1991

1993

1995

1997

1999

2001

2003

2005

2007

2009

2011

1.00

2.00

3.00

4.00

5.00

6.00

7.00 MULTIDISCIPLINARY JOURNALS

BASIC MEDICAL SCIENCES

BASIC LIFE SCIENCES

BIOMEDICAL SCIENCES

CLINICAL MEDICINE

ASTRONOMY AND ASTROPHYSICS

AGRICULTURE AND FOOD SCIENCE

CHEMISTRY AND CHEMICAL ENGINEERING

BIOLOGICAL SCIENCES

INSTRUMENTS AND INSTRUMENTATION

PHYSICS AND MATERIALS SCIENCE

ENERGY SCIENCE AND TECHNOLOGY

EARTH SCIENCES AND TECHNOLOGY

HEALTH SCIENCES

ENVIRONMENTAL SCIENCES AND TECHNOLOGY

ELECTRICAL ENGINEERING AND TELECOMMUNICATION

PSYCHOLOGY

MECHANICAL ENGINEERING AND AEROSPACE

COMPUTER SCIENCES

CIVIL ENGINEERING AND CONSTRUCTION

GENERAL AND INDUSTRIAL ENGINEERING

EDUCATIONAL SCIENCES

STATISTICAL SCIENCES

SOCIAL AND BEHAVIORAL SCIENCES, INTERDISCI-PLINARY

MANAGEMENT AND PLANNING

SOCIOLOGY AND ANTHROPOLOGY

INFORMATION AND COMMUNICATION SCIENCES

MATHEMATICS

LAW AND CRIMINOLOGY

ECONOMICS AND BUSINESS

LANGUAGE AND LINGUISTICS

POLITICAL SCIENCE AND PUBLIC ADMINISTRATION

HISTORY, PHILOSOPHY AND RELIGION

CREATIVE ARTS, CULTURE AND MUSIC

LITERATURE

26

Coverage issues

27

AU Moed, HF; Garfield, E. in WO

S

TI In basic science the percentage of 'authoritative' references decreases as bibliographies become shorter

SO SCIENTOMETRICS 60 (3): 295-303, 2004 Y

RF ABT HA, J AM SOC INF SCI T, v 53, p 1106, 2004 Y

GARFIELD, E. CITATION INDEXING, 1979 (BOOK!) N

GARFIELD E, ESSAYS INFORMATION S, v 8, p 403, 1985 N

GILBERT GN, SOC STUDIES SCI, v 7, p 113, 1977 Y

MERTON RK, ISIS, v 79, p 606, 1988 Y

ROUSSEAU R, SCIENTOMETRICS, v 43, p 63, 1998 Y

ZUCKERMAN H, SCIENTOMETRICS, v 12, p 329, 1987 Y

WoS Coverage = 5/7 = 71%

Not in WoS

WoS Coverage in 2010 across disciplines

• Black=Excellent coverage (>80%)

• Blue= Good coverage (between 60-80%)

• Green= Moderate coverage (but above 50%)

• Orange= Moderate coverage (below 50%, but above 40%)

• Red= Poor coverage (highly problematic, below 40%)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

BASIC LIFE SCIENCES (99,991)

BIOMEDICAL SCIENCES (105,156)

MULTIDISCIPLINARY JOURNALS (8,999)

CHEMISTRY AND CHEMICAL ENGINEERING (118,141)

CLINICAL MEDICINE (224,983)

ASTRONOMY AND ASTROPHYSICS (12,932)

PHYSICS AND MATERIALS SCIENCE (137,522)

BASIC MEDICAL SCIENCES (18,450)

BIOLOGICAL SCIENCES (60,506)

AGRICULTURE AND FOOD SCIENCE (26,709)

INSTRUMENTS AND INSTRUMENTATION (8,485)

EARTH SCIENCES AND TECHNOLOGY (33,160)

PSYCHOLOGY (24,244)

ENVIRONMENTAL SCIENCES AND TECHNOLOGY (42,705)

MECHANICAL ENGINEERING AND AEROSPACE (20,336)

HEALTH SCIENCES (29,213)

ENERGY SCIENCE AND TECHNOLOGY (15,021)

MATHEMATICS (27,873)

STATISTICAL SCIENCES (11,263)

GENERAL AND INDUSTRIAL ENGINEERING (8,756)

CIVIL ENGINEERING AND CONSTRUCTION (8,430)

ECONOMICS AND BUSINESS (16,243)

ELECTRICAL ENGINEERING AND TELECOMMUNICATION (...

MANAGEMENT AND PLANNING (7,201)

COMPUTER SCIENCES (23,687)

EDUCATIONAL SCIENCES (9,917)

INFORMATION AND COMMUNICATION SCIENCES (4,006)

SOCIAL AND BEHAVIORAL SCIENCES, INTERDISCIPLINARY...

SOCIOLOGY AND ANTHROPOLOGY (9,907)

LAW AND CRIMINOLOGY (5,299)

LANGUAGE AND LINGUISTICS (3,514)

POLITICAL SCIENCE AND PUBLIC ADMINISTRATION (6,423)

HISTORY, PHILOSOPHY AND RELIGION (11,753)

CREATIVE ARTS, CULTURE AND MUSIC (6,147)

LITERATURE (4,786)

Discipline(Publications in 2010)

% Coverage of references in WoS

Some clear ‘perversions’ of the system … ?

• “You call me, I call you”

• When time is passing by …

• Salami slicing to boost an academic career

• Multiple authorship (without serious contributing)

• Putting your name on everything your unit produces

• The role of self citations

• Jumping on hypes and fashionable issues