google scholar as a research and evaluation tool
DESCRIPTION
EC3metrics ha estado presente por primera vez en 2014 en la European Summer School for Scientometrics, la escuela de verano internacional que ofrece formación bibliométrica especializada cada año a 50 alumnos llegados desde todo el mundo.Nuestro compañero Álvaro Cabezas participó en dicho foro con una intervención acerca de las ventajas y limitaciones de Google Scholar, en una sesión dedicada a la evaluación de las Ciencias Sociales y Humanidades junto a Henk Moed, Philip Purnell, y Juan Gorráiz. En su intervención, Álvaro revisó los distintos productos de índole bibliométrica de Google Scholar mostrando sus puntos fuertes y débiles. Animó a los asistentes a experimentar con estos productos, si bien siendo conscientes de las precauciones que hay que tomar al usarlos con fines evaluativos.TRANSCRIPT
Google Scholar as a research and
evaluation toolevaluation tool
Álvaro Cabezas-Clavijo
INDEX
• Goals
• GS as a research tool– Search engine
– Alerts
– Updates
– Library + Cite
• GS as a tool for evaluating research• GS as a tool for evaluating research– GS Citations (or Profiles)
– GS Metrics
– Bibliometric analysis• Data manipulation
• Comparison to WOS/ Scopus
• Conclusions
�To show GS possibilities to do research
�To show GS possibilities to evaluate research
�To point out strengths and weaknesses of GS as a source to perform bibliometric analysis
GOALS
�To introduce Google Scholar as a valid source for carrying out research assessments (with caution!) in Social Sciences and Humanities (SSH)
GS AS A TOOL FOR RESEARCHERSGS AS A TOOL FOR RESEARCHERS
• GS is a widespread tool for doing research
• Comprehensive source of academic literature
• Widely known by scholars and students
• Ease of use (just like google)
GS AS A TOOL FOR RESEARCHERS
•• Free and fast
• Links to primary source (when it is freely available on the
web or can be accessed from our institution)
• Non-selective source (covers all material which meet the
technical requirements)
• No restrictions on language
• No restrictions on document type
• No discipline/areas bias
GS AS A TOOL FOR RESEARCHERS (II)
• No discipline/areas bias
• Estimated size is 100 million documents100 million documents (Web of Science
� 50 million; Scopus � 53 million)
GS AS A TOOL FOR RESEARCHERS (III)
• GS is not just an academic search engine but a “family” of tools for doing and evaluating research
GS AS A TOOL FOR RESEARCHERS (IV)
ALERTS UPDATES
LIBRARY
+ CITE
• Set up by user just clicking on the “Create alert” link when
performing a search
• Delivered by 2-4 days if there is fresh info to show
• Authors or thematic searches can be configured (but not for
journals)
ALERTS
journals)
1 2
3
• Configure an alert for citations to particular papers or authors
(in this case, if they have already set up a profile)
ALERTS
Following
Christian’s new
papers or
citations
• Alerts about papers of our interest automatically generated by Google
• You only need to set up a GS profile. Algorithm takes into account:– Who we cite in our papers
– Who cite us in their papers
UPDATES
– Who cite us in their papers
– Titles of our papers
– Co-authors
– Content
• A red bell indicates the number of new papers that might be of your interest
UPDATESSome paper recommendations
Ok, I have published about
these topics, so I may be
interested in these papers
Ok, this paper cites me so it should
be related to my research
• Kind of reference manager
• Save references in one click
• Arrange references by using labels
• Edit references & export them (but one by one!)
• Can’t import references from other bibliographic software
LIBRARY + CITE
• Can’t import references from other bibliographic software
•Cite easily from GS
results: formats �
MLA, APA, Chicago
•Get the formatted
reference and
copy/paste it into a
Bibliography.
• By default; 2 folders: My citations (actually My papers) and Cited by me
• You can set up other
LIBRARY + CITE
• You can set up other labels/folders and save new references from GS searches
GS as a tool for evaluating researchGS as a tool for evaluating research
• GS provides different tools which can help in assessments of individuals, journals and more
GS AS A TOOL FOR EVALUATORS (II)
PROFILES METRICS
SOURCE FOR
BIBLIOMETRIC ANALYSES
• Individual Web page with publication list and basic
bibliometric indicators
• Papers are ranked (by default) according to their number
of citations
• Also links to co-authors (if they have set up a profile) and
PROFILES
• Also links to co-authors (if they have set up a profile) and
keywordskeywordsRanking of researchers
by keywords/areas are
generated through this
classification
TIP: there is always a field where you
can be the most cited author! ☺
• Easy to set up and edit
• Strongly recommended for young researchers as it can
increase their visibility
• You will get an alert (maximum 3 times a week) every time
you are cited
PROFILES
• 72,579 researchers with profile (March 2012)
(Estimates by Ortega & Aguillo, 2013)
• GS profiles-derived products
– Some Spanish institutions are ranking their researchers
according to GSC (Univ. Granada, CSIC,…) � stimulus to set up
a profile?
COMMUNICATION: 886 RESEARCHERS
SCIENTOMETRICS: 265 RESEARCHERS
CSIC RANKING OF RESEARCHERS
ACCORDING TO GSC
• Easy to set up (just search for
your papers)
• Terrific tool for comparing
researchers within a field or
department
• Automatically updated basic
bibliometric data
PROFILES
• Data can be easily manipulated
(a researcher can self-claim
non-authored papers)
• Can stimulate vanity and ego
• Can generate no fair
comparisons (for example, bibliometric data
• No restrictions on source,
language or area
• Everyone can measure their
performance (and their
colleagues')
comparisons (for example,
researchers from different
areas in a single univ)
• Can generate unfair analysis by
non-bibliometric experts
STRENGTHS WEAKNESSES
• GS service devoted to measure journal’s impact according to their h-index (est. April 2012)
• Rankings are provided by language (nine languages) areas (8) and subcategories (313) (just for English-language journals) and journal title’s words (i.e, oncology)
• Subject/area rankings are limited to top 20 journals and language
METRICS
• Subject/area rankings are limited to top 20 journals and language rankings are limited to top 100 journals.
• Calculations take into account output for 5 years (currently 2008-2012) and citations received until july 2013. A new update is expected to be released in July 2014.
• Journals with fewer than 100 papers published in 5 years or with no citation (h-index=0) are excluded
• arXiv and other repositories’ collections are also included
• Free product to compare and rank journals
• We can get impact information about non-JCR journals and about national and SSH publications
• Transparency: citations for every paper that contributes to the h-index can be checked
• Methodological inconsistencies such as comparing journals from different areas
• Lack of proper bibliographic control (duplicates, “dirty” data,…)
• No selection criteria
• No action against data manipulation
METRICS
can be checked
• High correlation with JCR Impact factor (0.82)
• Simple to use and understand by any scholar
• Can be easily replicated
• No action against data manipulation
• No info about new updates (expected to be updated yearly)
• Just top results are presented
• H index favours journals with high production
STRENGTHS WEAKNESSES
• GS is estimated to contain about 100 million documents (87% of scholarly papers on the web) Khabsa & Giles (2014)
• So, it is the largest source for bibliometric analysis that we can use
SOURCE FOR BIBLIOMETRIC
ANALISYS
can use
• GS-based products can also be used to help with research assessment– Publish or Perish (software for bibliometric analysis)
– H-Index Scholar (ranking of spanish researchers)
– Google Chrome add-ons
– …
PUBLISH OR PERISH
H-INDEX SCHOLAR
GOOGLE CHROME ADD-ONS
• Many things to sort out (unknown coverage, duplicates, “dirty” data) but GS can be useful for:
�Non-covered areas by WoS…
�Non-English research…
SOURCE FOR BIBLIOMETRIC
ANALISYS
�Non-English research…
�…Which means basically Social Sciences and Humanities
�institutions and individuals which lack access to WoS/Scopus
GS data can be easily manipulated
• Just need to upload an “academic” paper to a institutional
repository, university webpage or similar
• “Academic” means a paper with title, authors, abstract and of
course, bibliographic references
• GS will count the citations from these fake papers, modifying • GS will count the citations from these fake papers, modifying
indicators of all cited scholars and journals.
Delgado, Robinson &
Torres (2014)
• Small diferences between rankings generated by GS, WoS and Scopus
• Intensive Care Medicine � Spearman correlation of 0.99 forjournals and 0.93 for top researchers
Comparison to Wos / Scopus
JOURNALSJOURNALS
Cabezas & Delgado (2013)
• Spearman correlation for Communication Journals: 0.895 (GSM-WoS); 0.879 (GSM-Scopus)
• Spearman correlation for Economics & Business Journals: 0.718 (GS-WoS)
JOURNALS
Delgado & Repiso (2013)
Harzing & der Wal (2009)
• Easy to use, fast and free
• Largest source for assessment
purposes
• Provides information about areas
and materials not covered by
• Lack of bibliographic control
• Data can be manipulated
• No possibilities to massive info
download
• Time-consuming, data needs to
SOURCE FOR BIBLIOMETRIC
ANALISYS
and materials not covered by
traditional bibliometric products
• Rankings are very similar to
those offered by WoS / Scopus
• Possibility to build new products
based on GS data
• Time-consuming, data needs to
be “cleaned”
• GS and their services could be
closed down in any moment
STRENGTHS WEAKNESSES
ConclusionsConclusions
• Bibliometricians need to find other products to complement WoS / Scopus-based evaluations (GS, ERIH, national products,…)
• What if subscriptions to traditional databases are cancelled?– Spain paying 4-6 million Euros to Thomson Reuters yearly for
national access to WoS from public research organizations
CONCLUSIONS
– Spain paying 4-6 million Euros to Thomson Reuters yearly for national access to WoS from public research organizations
• SSH deserve to be measured (prominent rankings such as Shanghai or Leiden ignore Humanities and a huge part of Social Sciences!)
• GS tools are far from perfect but provide us with some valuable information
• Bibliometricians take usually the “easy way”; only measure what can be easily measured with databases such as WoS or Scopus
• A challenge for bibliometricians is to get the most out of the existing tools to evaluate SSH (and to develop new ones!)
• GS can be used as a complementary or alternative source to
CONCLUSIONS
• GS can be used as a complementary or alternative source to shed some light on SSH
• Use with extreme caution, especially when assessing individuals
• Keep an eye on the future: Institutions might be demanding GS-based analysis soon.
Further readingsFurther readings
• Delgado-López-Cózar, E., & Cabezas-Clavijo, Á. (2013). Ranking journals: could Google Scholar Metrics be an alternative to Journal Citation Reports and Scimago JournalRank?. Learned publishing, 26(2), 101-113.
• Delgado López-Cózar, E., Robinson-García, N., & Torres-Salinas, D. (2014). The Google Scholar Experiment: how to index false papers and manipulate bibliometric indicators. Journal of the Association for Information Science and Technology, 65(3), 446-454.
• Harzing, A. W., & Van Der Wal, R. (2009). A Google Scholar h-index for journals: An
FURTHER READINGS
• Harzing, A. W., & Van Der Wal, R. (2009). A Google Scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60(1), 41-46.
• Khabsa, M., & Giles, C. L. (2014). The Number of Scholarly Documents on the PublicWeb. PloS one, 9(5), e93949.
• Ortega, J. L., & Aguillo, I. F. (2013). Institutional and country collaboration in an online service of scientific profiles: Google Scholar Citations. Journal of Informetrics, 7(2), 394-403.
• BLOG: Google Scholar Digest: analysis of papers dealing with GS http://googlescholardigest.blogspot.com.es/
• TWITTER: Follow the #googlescholar hashtag
• SOFTWARE: Publish or Perish http://www.harzing.com/pop.htm
• PRODUCTS:
FURTHER MATERIALS
• PRODUCTS:
• GOOGLE SCHOLAR CITATIONS http://scholar.google.com/citations
• GOOGLE SCHOLAR METRICS http://scholar.google.com/citations?view_op=top_venues&hl=en
Google Scholar as a research and
evaluation toolevaluation tool
Álvaro Cabezas-Clavijo
Thank you!