new developments in electronic publishing and bibliometrics
DESCRIPTION
New developments in electronic publishing and bibliometrics. Henk F. Moed CWTS, Leiden University, Netherlands Elsevier, Amsterdam, Netherlands. Contents. Contents. Journal impact measures are no good predictors of an individual paper’s actual citation impact. - PowerPoint PPT PresentationTRANSCRIPT
New developments in electronic publishing and bibliometrics
Henk F. Moed
CWTS, Leiden University, Netherlands
Elsevier, Amsterdam, Netherlands
Contents
1 Beyond journal impact factor and H-index
2 How useful are university rankings?
3 Why combine indicators and peer review?
4 Can indicators be manipulated?
5 Does Open Access lead to higher impact?
6 More downloads more citations?
Contents
1 Beyond journal impact factor and H-index
2 How useful are university rankings?
3 Why combine indicators and peer review
4 Can indicators be manipulated?
5 Does Open Access lead to higher impact?
6 More downloads more citations?
Journal impact measures are no good predictors of an individual paper’s
actual citation impact
Partly based on International Mathematical Union’s Report ‘Citation Statistics’ (2008)
Length boys vs. adults
0
5
10
15
20
25
30
35
0 15 35 55 75 95 115 135 155 175 195 215Length (cm)
% P
erso
ns
Boys (Meanlength=95 cm)Players (Meanlength=185 cm)
Citations to P-AMS vs. T-AMS
0
10
20
30
40
50
60
70
80
0 1 2 3 4 5 6 7 8Nr Cites
% P
aper
s
PAMS (JIF=0.43)
TAMS (JIF=0.85)
Normal vs. skewed distributions
0
5
10
15
20
25
30
35
0 15 35 55 75 95 115 135 155 175 195 215Length (cm)
% P
erso
ns
Boys (Meanlength=95 cm)Players (Meanlength=185 cm)
0
10
20
30
40
50
60
70
80
0 1 2 3 4 5 6 7 8Nr Cites
% P
aper
s
PAMS (JIF=0.43)
TAMS (JIF=0.85)
What is the probability that .......
a randomly selected boy is at least as tall as a randomly selected player?
Almost zero
62 %a randomly selected PAMS paper is cited at least as often as a randomly selected TAMS paper?
Probabilities still substantial for high JIF journals
‘Free’ citations
Thomson/JCR Journal Impact Factor
Citations to all docs
# Citable docs
Citable vs. non-citable docs
Citable documents “non-citable” documents
Articles Letters
Reviews Editorials
Discussion papers
The problem of “free” citations - 1
Cites
Docs + + + + +
+ + + + +
The problem of “free” citations - 2
Cites
Docs + +
+ + + + +
“Free” Citations
All three publication lists have a Hirsch Index of 5
30 P110 P2 8 P3 6 P4 5 P5 1 P6 0 P7
30 P110 P2 8 P3 6 P4 5 P5 4 P6 4 P7 4 P8 4 P9
100 P1 70 P2 8 P3 6 P4 5 P5 1 P6 0 P7
H=? H=? H=?5 5 5
123456789
Author 2Author 1 Author 3
Different bibliometric distributions
have the same H-Index
Indicators are becoming more informative
Feature Example
Embody ways to put numbers in context
Field-normalized citation measures
Take into account “who” is citing
Citations weighted with impact of citing source
Take into account relationship citing-cited author
Impact outside the own niche; multi-disciplinarity; bridging paradigms
Contents
1 Beyond journal impact factor and H-index
2 How useful are university rankings?
3 Why combine indicators and peer review
4 Can indicators be manipulated?
5 Does Open Access lead to higher impact?
6 More downloads more citations?
University ranking positions are primarily marketing tools,
not research management tools
Research assessment methodologies must take into account… [EC AUBR Expert Group]
1. Inclusive definition of research / output
2. Different types of research and its impacts
3. Differences among research fields
4. Type and mission of institution
5. Proper units of assessment
6. Policy context, purpose and user needs
7. The European dimension
8. Need to be valid, fair and practically feasible
Types of outputs (SSH)
Impacts Publication/text Non-publication
Scientific-scholarly
Journal paper; book chapter; monograph
Research data file; video of experiment
Educational Teaching course book; syllabus
Skilled researchers
Economic Patent Product; process; device; design; image
Cultural Newspaper article; Interviews; events; Performances; exhibits
Top-down institutional analysis
Select an institution’s papers using author addresses (incl. verification)
Categorize articles intoresearch fields
Calculate indicators
Compare with benchmarks
Bottom-up institutional analysis (CWTS)
Compile a list of researchers
Compile a list of publications per
researcher (incl. verification)
Aggregate researchers into groups, departments, fields, etc.
Calculate indicators;
compare with benchmarks
Secondary analyses of ‘ranking’ data are informative
Contents
1 Beyond journal impact factor and H-index
2 How useful are university rankings?
3 Why combine indicators and peer review
4 Can indicators be manipulated?
5 Does Open Access lead to higher impact?
6 More downloads more citations?
Case study: A national Research Council
• Proposals evaluated by committees covering a discipline
• Reports from external referees
• Committee members can be applicants
Affinity applicants – Committee
0 Applicants are/were not member of any Committee
1 Co-applicant is/was member of a Committee, but not of the one evaluating
2 First applicant is/was member of a Committee, but not of the one evaluating
3 Co-applicant is member of the Committee(s) evaluating the proposal
4 First applicant is member of the Committee(s) evaluating the proposal
For 15 % of applications an applicant is a member of the evaluating Committee (Affinity=3, 4)
0
10
20
30
40
50
60
70
% A
PP
LIC
AT
ION
S
AFFINITY APPLICANTS-COMMITTEE
Projects 63.2 10.2 11.5 5.9 9.1
0 1 2 3 4
Probability to be granted increases with increasing affinity applicants-Committee
30
40
50
60
70
80
% G
RA
NT
ED
AP
PL
ICA
TO
NS
AFFINITY APPLICANTS-COMMITTEE
Projects 37.0 46.9 60.1 62.6 74.0
0 1 2 3 4
Logistic regression analysis: Affinity Applicant-Committee has a significant effect
upon the probability to be granted
MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2,499) Source DF Chi-Square Prob ------------------------------------------------------------- INTERCEPT 1 18.47 0.0000 Publ Impact applicant 3 26.97 0.0000 ** Rel transdisc impact applicant 1 0.29 0.5926 Affinity applicant-Committee 2 112.50 0.0000 ** Sum requested 1 45.47 0.0000 ** Institution applicant 4 25.94 0.0000 ** LIKELIHOOD RATIO 199 230.23 0.0638
The future of research assessment exercises lies in the intelligent
combination of metrics and peer review
Contents
1 Beyond journal impact factor and H-index
2 How useful are university rankings?
3 Why combine indicators and peer review
4 Can indicators be manipulated?
5 Does Open Access lead to higher impact?
6 More downloads more citations?
Effects of editorial self-citations upon journal impact factors
[Reedijk & Moed, J. Doc., 2008]
• Editorial self-citations: A journal editor cites in his editorials papers published in his own journal
• Focus on ‘consequences’ rather than ‘motives’
Case: ISI/JCR Impact Factor of a Gerontology Journal (published in the journal itself)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
2000 2001 2002 2003 2004
IMPACT FACTOR YEAR
CIT
ES
PE
R 'C
ITA
BL
E' D
OC
Decomposition of the IF of a Gerontology journal
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
2000 2001 2002 2003 2004
IMPACT FACTOR YEAR
CIT
ES
PE
R 'C
ITA
BL
E' D
OC
Editorial self citations
Free citations
One can identify and correct for the following types of
strategic editorial behavior
• Publish ‘non-citable’ items• Publish more reviews• Publish ‘top’ papers in January• Publish ‘topical’ papers (with high short term
impact) • Cite your journal in your own editorials• Excessive journal self-citing
Contents
1 Beyond journal impact factor and H-index
2 How useful are university rankings?
3 Why combine indicators and peer review
4 Can indicators be manipulated?
5 Does Open Access lead to higher impact?
6 More downloads more citations?
Deposited in OA rep
(o)
Not depositedin OA rep.
(no)
Journal articles
Average Impact (CPPo)
Average Impact
(CPPno)
?><=
Three effects [Kurtz et al., 2005]
Open Access ArXiv increases access
Early View Articles appear earlier in ArXiv than in Publisher’s Website
Self-Selection (Quality bias)
Better authors use ArXiv
Authors deposit their best papers in arXiv
0
96 97 98 99 00 01 02 03 04 05
Publication Years
OA
Imp
ac
t a
dv
an
tag
e
Quality Bias: Better authors use ArXiv
EarlyView Effec
t
ArXiv papers appear earlier
ArXiv, Cond Mat Phys[Moed, JASIST 2007]
100
Age distribution of citations to Arxiv and non-ArXiv papers
0.00
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.10
0 6 12 18 24 30 36 42 48 54 60 66 72 78 84
Months after Publication Date
Cit
es
pe
r P
ap
er
in ArXiv-CM
Not in ArXiv-CM
3 per. Mov. Avg. (inArXiv-CM)3 per. Mov. Avg. (Notin ArXiv-CM)
Move curve by 6 months to the right
Early view effect: Citations to papers deposited in ArXiv-CM start about 6 months earlier
0.00
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.10
0 6 12 18 24 30 36 42 48 54 60 66 72 78 84
Months after Publication Date
Cit
es
pe
r P
ap
er
in ArXiv 6 monthstranslated
Not in ArXiv-CM
3 per. Mov. Avg. (inArXiv 6 monthstranslated)3 per. Mov. Avg. (Notin ArXiv-CM)
More research questions
• Early view effect also visible in a non-OA environment?
• Citation impact measured in biased sample?
Contents
1 Beyond journal impact factor and H-index
2 How useful are university rankings?
3 Why combine indicators and peer review
4 Can indicators be manipulated?
5 Does Open Access lead to higher impact?
6 More downloads more citations?
Downloads vs. Citations
More downloads more citations
or
More citations more downloads?
Relation between citations and internet hits for 153 papers in volume 318 of the BMJ (1999)
Figure 1 from: Perneger, TV. BMJ. 2004, 329 (7465): 546–547. Relation between online “hit counts” and subsequent citations: prospective study of research papers in the BMJ
Analogy Model
Formal use (e.g., SCI) Informal use (e.g.,SD)
(Collections of) publishing authors
(Collections of) users
Citing a document Retrieving the full text of a document
Article User session
Author’s institutional affiliation
User’s account name
Number of times cited Number of times retrieved as full text
Age distribution downloads vs. citations[Tetrahedron Lett, ScienceDirect; Moed, JASIST, 2005]
0
4
8
12
16
20
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
AGE (MONTHS)
%
SD USESCITATIONSDownloads
Citations%
Age (months)
Ageing downloads vs. citations: Two factor vs. single factor model
0.01
0.1
1
10
100
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88
Age (months)
Us
es
DownloadsObserved
DownloadsComputed
DownloadsSingularPoints
CitationsObserved
CitationsComputed
%
Age (months)
Downloads
Citations
Citations lead to downloads[Moed, J. Am Soc Inf Sci Techn, 2005]
1
10
100
1000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
AGE PAPER A (MONTHS)
DO
WN
LO
AD
S
A
B (B cites A)
C (C cites A and B)
PaperA published
Paper B published;it cites A
Download of A increases
Paper C published;it cites A and B
Rank correlations between downloads and citations
Variables Spearman
R
Downloads vs. citations 0.22
‘Later’ (>3 months) downloads vs. citations
0.33
‘Initial’ (<3 months) downloads vs. citations
0.11
Conclusions
• Positive correlation between downloads and citations partly due to the effect of citations upon downloads
• ‘Initial’ downloads and citations hardly correlate, and relate to distinct phases in processing relevant scientific information
• ‘Later’ downloads and citations show statistically similar properties of ageing and frequency distribution
Downloads and citations relate to distinct phases in
scientific information processing
.... but (many) more cases must be studied
Thank you for your attention!
References• Eysenbach, G. (2006). Citation Advantage of Open Access Articles. PLOS Biology, 4, 692–698.• Garfield, E. (1972). Citation Analysis as a tool in journal evaluation. Science, 178, 471–479.• Garfield, E. (1979). Citation Indexing. New York: Wiley.• Yassine Gargouri, Chawki Hajjem, Vincent Lariviere, Yves Gingras, Les Carr, Tim Brody, Stevan Harnad
(2010). Self-Selected or Mandated, Open Access Increases Citation Impact for Higher Quality Research arXiv:1001.0361v2 [cs.CY]
• Harnad, S., & Brody, T. (2004). Comparing the impact of open access (OA) vs. Non-OA articles in the same journals. D-Lib Magazine, 10, Nr 6.
• Kurtz, M.J., Eichhorn, G., Accomazzi, A., Grant, C., Demleitner, M., Henneken, E., & Murray, S.S. (2005). The effect of use and access on citations. Information Processing & Management, 41, 1395–1402.
• Moed, H.F. (2005). Citation Analysis in Research Evaluation. Dordrecht (Netherlands): Springer. ISBN 1-4020-3713-9, 346 pp.
• Moed, H.F., Glänzel, W., and Schmoch, U. (2004) (eds.). Handbook of Quantitative Science and Technology Research. The Use of Publication and Patent Statistics in Studies of S&T Systems. Dordrecht (the Netherlands): Kluwer Academic Publishers, 800 pp.
• Moed, H.F. (2005). Statistical relationships between downloads and citations at the level of individual documents within a single journal. Journal of the American Society for Information Science and Technology 56, 1088-1097.
• Moed, H.F. (2007). The effect of “Open Access” upon citation impact: An analysis of ArXiv’s Condensed Matter Section. Journal of the American Society for Information Science and Technology 58, 2047-2054.
• Moed, H.F. (2009). New developments in the user of citation analysis in research evaluation. Archivum Immunologiae et Therapiae Experimentalis (Warszawa) 17, 13-18.
• Reedijk, J., Moed, H.F. (2008). Is the impact of journal impact factors decreasing? Journal of Documentation 64, 183-192.
ISI/JCR Journal Impact Factor of journal J for year T
Citations in year T to items published in J in years T-1 and T-2
÷Number of “citable” items published in J in
years T-1 and T-2