on the limits of bibliometrics
DESCRIPTION
Apesar de seu uso generalizado, os métodos bibliométricos têm várias limitações, especialmente quando se trata da análise dos países em desenvolvimento. Esta apresentação irá recuperar estas limitações (cobertura parcial da literatura nacional em países de língua não inglesa, baixa cobertura de livros, etc.), bem como fornecer dados sobre a sua extensão nas diferentes disciplinas. Também serão detalhados as limitações do Fator de Impacto e os argumentos contra a sua utilização em avaliação da pesquisa. Despite their widespread use, bibliometric methods have several limitations, especially when it comes to the analysis of developing countries. This presentation will recall these limitations (partial coverage of national literature for non-English speaking countries, weak coverage of books, etc.) as well as provide data on their extent in the different disciplines. It will also detail the drawbacks of the Impact Factor and argue against its use in research evaluation. A pesar de su uso generalizado, los métodos bibliométricos tienen varias limitaciones, especialmente cuando se trata del análisis de los países en desarrollo. Esta presentación recordará estas limitaciones (cobertura parcial de la literatura nacional de los países de habla no inglesa, la debilidad de la cobertura de libros, etc), así como proporcionar datos sobre su extensión en las distintas disciplinas. También describirá pormenorizadamente los inconvenientes del Factor de Impacto y argumentar en contra de su uso en la evaluación de la investigación.TRANSCRIPT
Faculté des arts et des sciences SciELO15 anos
1/15
Faculté des arts et des sciencesÉcole de bibliothéconomie et des sciences de l'information
On the Limits of Bibliometrics
Vincent LarivièreCanada Research Chair on the transformations
of scholarly communicationUniversité de Montréal
SciELO 15 anos
24 Outubro 2013
Faculté des arts et des sciences SciELO15 anos
2/15
Outline of the talk
- Technical limits
- Coverage
- Limits of indicators
- Impact Factor and prediction of citations
- Citation window
- Adverse consequences
Faculté des arts et des sciences SciELO15 anos
3/15Technical limits
- No database is covers all of the published literature.
- Particularly true in the social sciences and humanities.
- Local journals are not indexed by major databases and hence, local topics are not well covered
- No books were indexed (until recently).
Faculté des arts et des sciences SciELO15 anos
4/15
Percentage of references made to journal articles in the natural and medical sciences (Larivière et al. 2006)
55%
60%
65%
70%
75%
80%
85%
90%
95%
1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000
Year
Average SCI Biomedical Research Clinical MedicinePhysics Biology Earth and Space SciencesChemistry Psychology MathematicsEngineering
Faculté des arts et des sciences SciELO15 anos
5/15Percentage of references made to journal articles in the social sciences and humanities (Larivière et al. 2006)
Faculté des arts et des sciences SciELO15 anos
6/15Proportion of references made to conference proceedings in the natural and medical sciences (Lisée
et al. 2008)
0% 5% 10% 15% 20%
Applied Physics
Metals & Metallurgy
General Engineering
Optics
Chemical Engineering
Library & Information Science
Human Factors
Acoustics
Mechanical Engineering
Industrial Engineering
Aerospace Technology
Miscellaneous Engineering & Technology
Nuclear Technology
Civil Engineering
Electrical Engineering & Electronics
Computers
Faculté des arts et des sciences SciELO15 anos
7/15Coverage of journals as a function of the
language of publication (Archambault et al. 2006)
NSE SSH
Thomson ISI Ulrich Difference Thomson ISI Ulrich Difference
English 89% 78% 13% 90% 75% 20%
Czech 0.04% 0.3% -85% 0.2% 0.2% 8%
Russian 0.5% 0.9% -48% 0.3% 0.4% -24%
French 3.3% 3.4% -3% 3.2% 4.4% -26%
Multiple languages 0.2% 0.2% -14% 0.3% 0.5% -45%
Dutch 2.2% 2.2% 0% 1.3% 2.6% -48%
German 3.2% 3.9% -18% 3.0% 5.8% -50%
Japanese 0.4% 1.7% -74% 0.2% 0.6% -64%
Swedish - 0.1% -100% 0.1% 0.4% -69%
Spanish 0.6% 2.6% -75% 0.9% 3.0% -69%
Italian 0.1% 0.8% -83% 0.2% 1.1% -80%
Danish 0.04% 0.1% -50% 0.1% 0.3% -83%
Portuguese 0.1% 0.7% -85% 0.1% 1.0% -86%
Chinese 0.3% 2.4% -88% 0.04% 1.2% -96%
Polish 0.05% 0.7% -92% - 0.9% -100%
Arabic - 0.1% -100% - 0.3% -100%
Turkish 0.01% 0.2% -95% - 0.1% -100%
Other 0.3% 1.6% -80% 0.3% 2.4% -87%
Language
Faculté des arts et des sciences SciELO15 anos
8/15
Median age of cited literature(Archambault et Larivière, 2010)
0
2
4
6
8
10
12
14
16
1980 1985 1990 1995 2000
Âge
méd
ian
de la
litt
érat
ure
cité
e
Année
Arts ethumanités
Sciencessociales
Sciencesnaturelleset génie
Médecine
Faculté des arts et des sciences SciELO15 anos
9/15
How to correct (some of) these limits
- Creation of book databases
- Book Citation Index (but it still has an anglo-american bias!)
- Triangulate data sources
- Scopus vs. Thomson vs. Google Scholar
- Creation of local databases
- SciELO in Brazil
- Érudit au Québec
Faculté des arts et des sciences SciELO15 anos
10/15
Impact factor and its limits
Cannot be compared across disciplines
Skewness of citation distributions
Assymetry between numerator and denominator
Short citation window – especially for SSH
Includes journal self-citations
Is less and less correlated with citations received by papers
Faculté des arts et des sciences SciELO15 anos
11/15
Different citation practices: mean citation rates of papers after 2 and 10 years, 1900-2005
0
1
2
3
4
5
6
7
1900 1920 1940 1960 1980 2000
Mean - 2 year citation window
MED
NSE
SS
0
2
4
6
8
10
12
14
16
18
20
22
24
1900 1920 1940 1960 1980 2000
Mean - 10 year citation window
MED
NSE
SS
Faculté des arts et des sciences SciELO15 anos
12/15
Weakening relationship between Impact Factor and citations (Lozano et al. 2012)
y = 0.0011x - 1.9418r2 = 0.4666
y = 0.003x - 5.7759r2 = 0.579
y = -0.0049x + 9.9863r2 = 0.4788
0,00
0,05
0,10
0,15
0,20
0,25
0,30
0,35
1900 1920 1940 1960 1980 2000 2020
Annu
al c
oeffi
cient
of d
eter
min
atio
n (r2
)
Faculté des arts et des sciences SciELO15 anos
13/15
Decline of top journals (Larivière et al. 2013)
Faculté des arts et des sciences SciELO15 anos
14/15
Adverse effect
- Change of research topics
Type of journal Brazil in abstractBrazilian Journals 31%USA Journals 15%Other Journals 22%
Faculté des arts et des sciences SciELO15 anos
15/15
Archambault, É., Vignola-Gagné, É., Côté., Larivière, V. and Y. Gingras (2006). Benchmarking scientific output in the social sciences and humanities: The limits of existing databases, Scientometrics, 68(3): 329-342.
Larivière, V., Archambault, É, Gingras, Y. and É. Vignola-Gagné (2006). The place of serials in referencing practices: Comparing natural sciences and engineering with social sciences and humanities, Journal of the American Society for Information Science and Technology, 57(8): 997-1004.
Larivière, V., Archambault, É., Gingras, Y. (2008) Long-term variations in the aging of scientific literature: from exponential growth to steady-state science (1900-2004). Journal of the American Society for Information Science and Technology, 59(2): 288–296.
Larivière, V., Gingras, Y. and É. Archambault (2006). Canadian Collaboration Networks: A Comparative Analysis of the Natural Sciences, Social Sciences and the Humanities, Scientometrics, 68(3): 519-533.
Larivière, V. , Lozano, G.A., Gingras, Y. (2013). Are elite journals declining? Forthcoming in the Journal of the American Society for Information Science and Technology.
Lozano, G.A., Larivière, V., Gingras, Y. (2012) The weakening relationship between the Impact Factor and papers’ citations in the digital age. Journal of the American Society for Information Science and Technology. 63(11): 2140-2145.
Lisée, C., Larivière, V., Archambault, É. (2008) Conference proceedings as a source of scientific information: A bibliometric analysis. Journal of the American Society for Information Science and Technology, 59(11): 1776–1784.
Bibliography