open access and research evaluation thursday 19th october 14.15-15.45 session 4: quality assessment...

“Open access and research evaluation”Thursday 19th October14.15-15.45 Session 4: Quality Assessment

Open Scholarship 2006: New Challengesfor Open Access RepositoriesThe University of Glasgow 18-20 October 2006

Institut für Forschungsinformation und Qualitätssicherung

Hornbostel

19.10.2006

Institute for Research Information and Quality Assurance

Stefan HornbosteliFQ Institut für Forschungsinformation und QualitätssicherungInstitute for Researchinformation and Quality AssuranceGodesberger Allee 90D-53175 Bonn www.forschungsinfo.de

http://www.forschungsinfo.de/


Hornbostel

19.10.2006

Relations between OA and research evaluation

political / rethorical function to spread OA(to maximize the impact of their articles authors should archive!)

OA offers a better way to evaluate research performance?(Compensation of shortcomings of traditional bibliometric Indicators)

OA makes it possible to construct new indicators to measure a completely different dimension (usage, reading, …)

OA changes scientific behaviour (more easily accessibility, sooner availability, quick circulation, high quality).


Hornbostel

19.10.2006

Relations between OA and research evaluation (1)

political / rethorical function to spread OA(to maximize the impact of their articles authors should archive!)

Research evaluation practices are used to convince authors of OA advantages (higher Citationrates, higher attention)

The analysis covers 1,494 publication venues containing at least 5 online and 5 offline articles. For 90% of venues, online articles are more highly cited on average. On average there are 336% more citations to online articles compared to offline articles.

Lawrence, S. "Online or Invisible" Nature 411 (6837): 521, 2001.


Hornbostel

19.10.2006

„Mandated online RAE CVs Linked to University Eprint Archives“,

“The Funding Councils should mandate that ……. the full text of every refereed research paper, publicly self-archived in the university's online Eprint Archive and linked to the CV for online harvesting, scientometric analysis and assessment.

This will

increase the uptake and impact of UK research output, by increasing its visibility, accessibility and usage, and

set an example for the rest of the world that will almost certainly be emulated, in both respects: research assessment and research access.”

Harnad, S., Carr, L., Brody, T. and Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives. Ariadne 35.


Hornbostel

19.10.2006

“The proportion of articles for which their authors provide OA is likely to increase dramatically now, in part because of the mounting evidence for the impact advantage OA confers.” Brody, T. and Harnad, S. (2004) Comparing the Impact of Open Access (OA) vs. Non-OA Articles in the Same Journals. D-Lib Magazine 10(6).


Hornbostel

19.10.2006

For Example: Cream of Science

•Target“demonstrat(ing) that scholars are willing to deposit their materials in a repository, thereby also increasing the awareness of other scholars”

• Method“DARE partners selected ten of their prominent scientists and made their complete publication list, with as much full text available as possible” • Selection Criteria“there was no objective criteria to define 'prominent',…but most of the DARE partner used a formal method; selection by the Executive Board and/or using a letter from the Dean to invite academics to be part of Cream”Martin Feijen and Annemiek van der Kuil 2005: A Recipe for Cream of Science: Special Content Recruitment for Dutch Institutional Repositories', Ariadne Issue 45.

“Researchers are enthusiastic about the initiative. A number of them already refer to www.creamofscience.org on their own websites for a complete overview of their work. They regard Cream of Science as a hallmark of quality.” Dare.net 17.05.2005

http://www.creamofscience.org/


Hornbostel

19.10.2006

Kurtz et al. claim that there are (at least) three possible, non-exclusive effects that cause the effect of higher citation rates.

Open Access Postulate (OA): Because of free, unrestricted access, papers are read more easily and therefore get cited more frequentlyEarly Access Postulate (EA): Papers offered as e-print are available sooner and therefore gain primacy and additional time in press, and therefore they get cited more oftenSelf-selection Bias Postulate (SB) or (Quality Postulate) : The most important -- and therefore most citable -- papers are posted on the Internet

Kurtz, Michael J., Eichhorn, Guenther, Accomazzi, Alberto, Grant, Carolyn, Demleitner, Markus, Henneken, Edwin, Murray, Stephen S. (2005). “The Effect of Use and Access on Citations”, Information Processing and Management, Vol. 41, Issue 6, p. 1395-1402.

OA a causal factor ?


Hornbostel

19.10.2006

• no relationship between fulltext arXiv downloads and the number of article citations until an article has been downloaded more than 400 times • consistent with the detailed work of Henk Moed on the journal Tetrahedron Letters, which suggests that a relatively small group of both highly cited and frequently downloaded papers are responsible for the weak, though statistically significant, correlation between downloads and citations

Philip M. Davis / Michael J. Fromerth (2006): Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles?

Open Access Postulate


Hornbostel

19.10.2006

Early Access Postulate

We have failed to find support for the Early View postulate and have provided evidence that many highly-cited articles were deposited into the arXiv after formal publication.

Partial regression plot of prepublication days on citation, controlling for the number of days since publication and the effect of the journal. Articles deposited in the arXiv after formal publication are highlighted red.

Philip M. Davis / Michael J. Fromerth (2006): Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles?


Hornbostel

19.10.2006

• approximately 15% of the articles in a journal collect 50% of the citations

• If authors are indeed depositing their best papers, Open Access citation advantage may be an artifact, not the cause of a citation advantage

Self-selection Bias Postulate (SB) or Quality Postulate


Hornbostel

19.10.2006

Problems: Open Access (green and gold), Open Archive, Self Archiving may produce different quality distributions J. Testa and M.E. McVeigh. 2004.“The Impact of Open Access Journals.” A Citation Study from Thomson ISI, www.isinet.com/oaj .

http://www.isinet.com/oaj


Hornbostel

19.10.2006

OA offers a better way to evaluate research performance?(Compensation of shortcomings of traditional bibliometric Indicators)

“These new possibilities may be interesting seen in combination with the known inherent bias of research evaluation using citation analysis. …..”Tove Faber Frandsen (2006): Open access-based resources in research evaluation

The Funding Councils should mandate that … all UK research-active university staff must maintain: .... (II) the full text of every refereed research paper, publicly self-archived in the university's online Eprint Archive …..

This will give the UK Research Assessment Exercise (RAE) far richer, more

sensitive and more predictive measures of research productivity and impact, for far less cost and effort (both to the RAE and to the universities preparing their RAE submissions)…

Harnad, S., Carr, L., Brody, T. and Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives. Ariadne 35.


Hornbostel

19.10.2006

ISI / WOS coverage by main field

EXCELLENT (> 80%)

Good (60-80%) Good (40-60%) MODERATE (<40 %)

Biochem & Mol Biol Appl Phys & Chem Mathematics Other Soc Sci

Biol Sci ~ Humans Biol Sci ~ Anim &Plants

Economics Humanities & Arts

Chemistry Psychol & Psychiat Engineering

Clin Medicine Geosciences

Physics & Astron Soc Sci ~ Medicine & Health

Quelle: Henk v. Moed, Evaluation of Research Performance and Funding Programme in Social Sciences; Tagung: Norface Workshop on Research Programme Development and Management 7.02.06, Bonn: http://www.norface.org/norface/files/file_20060207014936Henk%20Moed.pdf

http://www.norface.org/norface/files/file_20060207014936Henk%20Moed.pdf


Hornbostel

19.10.2006

Comparison of Mean Citation Rates Between Freely Available Articles and Those That Are Not Freely Available

Discipline Mean(open)

Mean Standard Error(open)

Mean(not open)

Mean Standard Error(not open)

Difference in Means

Philosophy 1.60 0.491 1.10 0.230 .500

Political Science 2.20 0.477 1.18 0.353 1.016

Electronical and electronic engineering

2.35 0.449 1.56 0.275 .798

Mathematics 1.60 0.270 0.84 0.230 .762

Antelman, Kristin (2004) Do Open Access Articles Have a Greater Research Impact?. College & Research Libraries News 65(5):pp. 372-382.


Hornbostel

19.10.2006

In electrical engineering, computer science and system engineering, 34 % of the respondents know of this form of publication; the highest percentage here is among computer scientists, at 51 %. In mechanical engineering and production engineering, 31 % of the scientists said they knew about open access journals, compared to just 13 % in the area of heat energy technology and process engineering. In the areas of civil engineering and architecture this type of publication is completely unknown.

Deutsche Forschungsgemeinschaft (2005): Publishing Strategies in Transformation?Results of a study on publishing habits and information acquisition with regard to open access


Hornbostel

19.10.2006


In the humanities and social sciences, it is mostly social and behavioural scientists who use the option of secondary open access publication for their academic studies. While they make around 9% of the articles published in journals available for free on the internet, this is done by only 3% of the humanities scholars surveyed.

In the engineering sciences it is the computer scientists who stand out, with two-thirds of their articles in conference proceedings, 46 % of their journal articles, almost 42 % of their contributions to edited volumes and nearly 24% of their monographs republished on the internet.


Hornbostel

19.10.2006



Hornbostel

19.10.2006

• Richer cross-repository services, that is services that overlay multiple repositories

• Cross-repository scholarly communication workflows

Examples of such value-chains include:

•The transfer of (parts of) digital objects from digital repositories to parties that provide discovery-oriented services over (parts of) the digital objects.

•The transfer of digital objects from digital repositories to parties that provide value-added services over the

digital objects.

• The introduction of natively machine-readable and machine-actionable bibliographic citations.

• The re-use of datasets (from different repositories) as the basis for building a new dataset or for writing a publication.

AUGMENTING INTEROPERABILITY ACROSS SCHOLARLY REPOSITORIESA meeting sponsored and supported by Microsoft, the Andrew W. Mellon Foundation, the Coalition for Networked Information, the Digital Library Federation, and the Joint Information Systems Committee.April 20 – 21, 2006, New York, NY

Ithaca, NY and Los Alamos, NM - The Open Archives Initiative (OAI), with the generous support of the Andrew W. Mellon Foundation, announces a new effort as part of its mission to develop and promote interoperability standards that aim to facilitate the efficient dissemination of content. Object Reuse and Exchange (ORE) will develop specifications that allow distributed repositories to exchange information about their constituent digital objects. These specifications will include approaches for representing digital objects and repository services that facilitate access and ingest of these representations. The specifications will enable a new generation of cross-repository services that leverage the intrinsic value of digital objects beyond the borders of hosting repositories.


Hornbostel

19.10.2006

OA changes scientific behaviour (more easily accessibility, sooner availability , quick circulation, high quality)

Scholarly communication has changed rapidly over the last decade as scientists now publish their findings in e.g. open access journals and preprint servers. Bauer & Bakkalbasi (2005)

The communication system is changing primarily due to the use of new technology. Friend (2004)


Hornbostel

19.10.2006

But: only c. 20% of the number of papers published annually are open access


Hornbostel

19.10.2006

Poeschl,U. 2006,Peer Review Revisited, 16.05.2006, Ms.


Hornbostel

19.10.2006

OA makes it possible to construct new indicators to measure a completly different dimension (usage, reading, …)

Web indicators based on usage data are a way to obtain unique information of the utility and scholarly acceptance of OA research output, but these new user oriented web indicators are still in an early experimental phase. The rationale of these approaches is strong but standardization of web metrics is at the beginning. …. there is still much to be done in evaluation of OA research. In addition we need further approaches to build robust measures and metrics especially for OA publications/documents. Following the thesis “science is turning to e-science” … we conclude e-science will need new stable e-indicators or a combination with traditional indicators to appropriately take e-only research output into consideration.

Philipp Mayr (2006): Constructing experimental indicators for Open Access documents.


Hornbostel

19.10.2006

First, journal relationships in the RGN network seem to be valid and representative of the community whose downloads have shaped the network. Second, structural journal impact metrics derived from the RGN deviate strongly from the ISI IF. Third, the applied structural impact metrics correlate strongly with the ISI IF when calculated over the AGN, indicating they do validly operationalize journal impact, if we honor the assumption that the ISI IF does. Fourth, the AGN and RGN networks overlap to some degree, but exhibit striking differences.

Johan Bollen / Herbert Van de Sompel / Joan A. Smith / Rick Luce (Los Alamos National Laboratory) 2005: Toward alternative metrics of journal impact:A comparison of download and citation data.

The rendered journal impact rankings were compared to those derived from the Reader Generated Network, and the ISI IF. Our results indicate the following:

Darmoni, Roussel, Benichou, Thirion, and Pinhas (2002) compare journal usage frequency to the ISI IF for a medical DL collection. They define a “Reading Factor” (RF) which consists of the ratio of a particular journal’s download frequency to the total downloads of all journals as recorded in the DL’s logs. The authors report a low and statistically insignificant correlation between the observed RF and the ISI IF for the same set of journals.


Hornbostel

19.10.2006

Standards!

OA research evaluation should follow the classical methods of bibliometrics:

Measuring Research activityP amount of PublicationsP% Percentage of worldwide Publications (international comparison) or national Publications (national comparison)

Measuring ImpactC amount of citationsPp% percentage of publications not cited until nowCmax Maximum of CitationsC/P Citation RateC/P/FCSm (mean field citation score) Relative Citation Rate (related to field)(C/P)/JCSm (mean journal citation score) Relative Citation Rate (related to journal)


Hornbostel

19.10.2006

Thank you!


Hornbostel

19.10.2006

Difficulties of OA research evaluation

Only fits to publications in traditional databases - especially ISI

Difficult for publications which are not covered by ISI

Missing subfield information

Metadata

No standardised metadata

No gold standard for the evaluation of open access papers

No best practices of hundred percent covering

Units

Main units for OA research evaluation are inexplicit

Different kinds of publications

Different motives for OA publishing -> bias!


Hornbostel

19.20.2006

It was found that during the first 3 months after an article is cited, its number of downloads increased 25% compared to what one would expect this number to be if the article had not been cited. Moreover, more downloads of citing documents led to more downloads of the cited article through the citation. An analysis of 1,190 papers in the journal during a time interval of 2 years after publication date revealed that there is about one citation for every 100 downloads. A Spearman rank correlation coefficient of 0.22 was found between the number of times an article was downloaded and its citation rate recorded in the SCI. When initial downloads - defined as downloads made during the first 3 months after publication - were discarded, the correlation raised to 0.35. However, both outcomes measure the joint effect of downloads upon citation and that of citation upon downloads. Correlating initial downloads to later citation counts, the correlation coefficient drops to 0.11. Findings suggest that initial downloads and citations relate to distinct phases in the process of collecting and processing relevant scientific information that eventually leads to the publication of a journal article.

Statistical relationships between downloads and citations at the level of individual documents within a single journal

Henk F. Moed Journal of the American Society for Information Science and Technology

Volume 56, Issue 10 , Pages 1088 - 1097Published Online: 31 May 2005


Hornbostel

19.20.2006

Theseresults furthermore raise questions regarding the validity of the ISI IF as the soleassessment of journal impact, and suggest the possibility of devising impact metricsbased on usage information in general.

Toward alternative metrics of journal impact:A comparison of download and citation data.Johan BollenDepartment of Computer Science, Old Dominion University, 4700 Elkhorn Ave.,Norfolk VA 23529Herbert Van de SompelResearch Library, Los Alamos National Laboratory, Los Alamos, NM, 87554Joan A. SmithDepartment of Computer Science, Old Dominion University, 4700 Elkhorn Ave.,Norfolk VA 23529Rick LuceResearch Library, Los Alamos National Laboratory, Los Alamos, NM, 87554

Preprint submitted to Elsevier Science 17 May 2006

Similarly, Darmoni, Roussel, Benichou, Thirion, and Pinhas (2002) comparejournal usage frequency to the ISI IF for a medical DL collection. They definea “Reading Factor” (RF) which consists of the ratio of a particular journal’sdownload frequency to the total downloads of all journals as recorded in theDL’s logs. The authors report a low and statistically insignificant correlationbetween the observed RF and the ISI IF for the same set of journals. Theseresults show that journal download frequency within a local DL communitydoes not correspond to the ISI IF, which raises questions regarding the ISIIF’s validity as the sole indicator of Ig among a specific community of readers.4

open access and research evaluation thursday 19th october 14.15-15.45 session 4: quality assessment...

Documents

research access

impact of open access

online articles

open access postulate

nonoa articles

impact advantage oa

research assessment

research information