Alternative Peer Review:Quality Management
for 21st Century Scholarship
Gerry McKiernan
Science and Technology Librarian
and Bibliographer
Iowa State University Library
Ames IA
http://www.public.iastate.edu/~gerrymck/APR.ppt
Workshop on Peer Review in the Age of Open Archives
International School for Advanced StudiesInterdisciplinary Laboratory
Trieste, ItalyMay 23-24, 2003
THANK YOU!Workshop Advisory Board [(Marco Fabbrichesi (INFN/SISSA Italy), Stevan Harnad (University of Southampton, UK), Stefano Mizzaro (University of Udine, Italy) and Corrado Pettenati (CERN Library, Geneva, Switzerland)]Iowa State University, Faculty Senate, Committee on Recognition and DevelopmentEuropean CommissionIowa State University LibraryHeike Kross, Ph.D.
DISCLAIMER (1)The screen prints selected for this presentation are for educational
purposes and their inclusion does not constitute an endorsement of an associated product, service,
place, or institution.
DISCLAIMER (2)The views and opinions expressed
in this presentation are those of the presenter and do not constitute
an endorsement by Iowa State University or its Library.
NOTICE
No editors, authors, or referees were harmed in the preparation of
this presentation.
http://www.sciencecartoonsplus.com/
Giuseppe De Nittis (1846-1884)The Macchiaioli / Italian Impressionists
Campo di Biche (1875)
PEER REVIEW: DEFINITION
“Peer review is the assessment by an expert of material submitted for publication.”
Carin M. Olson, “Peer Review of the Biomedical Literature,” American Journal of Emergency Medicine 8 no.4 (July 1990): 356-358.
PEER REVIEW: PURPOSES
Carin M. Olson, “Peer Review of the Biomedical Literature,” American Journal of Emergency Medicine 8 no.4 (July 1990): 356-358.
Peer review helps to ensure that published research is:
Important Original
Timely Technically-reliable
Internally-consistent Well-presented
Benefited from guidance by experts
PEER REVIEW: STRENGTHS
Anne C. Weller, Editorial Peer Review: Its Strengths and Weaknesses. (Medford, NJ: Information Today, 2001).
The underlying strength of peer review is“…the concerted effort by large numbers of researchers and scholars who work to assure that valid and valuable works are published and conversely to assure that invalid or non-valuable works are not published … .”
“Houston, We Have a Problem!”
PEER REVIEW: PROBLEMSSubjectivity
Bias
Abuse
Detecting defects
Fraud and Misconduct
DelayFytton Rowland, “The Peer-Review Process,”
Learned Publishing 15 no. 4 (October 2002): 247-258.
Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf
SUBJECTIVITYSummary rejections by editor without sending the paper to referees
Choice of referees by the editor (choosing for example, a known harsh referee for a paper the editor wishes to see rejected)
BIASDiscrimination against authors because of their nationality, native language, gender or host institution
Situations where author and referee are competitors in some sense, or belong to warring schools of thought
ABUSEToo many articles out of one piece of research, or duplicate publicationIntellectual theft: omission or downgrading of junior staff by senior authorsPlagiarism (stealing others yet unpublished work that has been sent for review)Delaying publication of potentially competing research
DETECTING DEFECTSIdentification of factual errors within submission
FRAUD and MISCONDUCTFabrication of results
Falsification of data
False claim of authorship for results
DELAY“There is much muttering about publication delay, a real enough problem, especially in paper publication, but peer review itself is often responsible for as much of the delay as the paper publication and distribution process itself.”
Stevan Harnad
Stevan Harnad, “Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals, in Scholarly
Publication: The Electronic Frontier, edited by Robin P. Peek and Gregory B. Newby (Cambridge MA: MIT Press, 1996).
http://www.ecs.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html
http://www-marketing.wharton.upenn.edu/ideas/pdf/peerpaper.pdf
“Peer review is slow, expensive, profligate of academic time, highly subjective, prone to bias,
easily abused, poor at detecting gross defects, and almost useless in detecting fraud.”
Richard Smith, “Opening Up BMJ Peer Review,”
BMJ 318 (7175) (January 2 1999): 4-5
Richard SmithEditor, BMJ
Stephen Lock, A Difficult Balance: Editorial Peer Review in Medicine (Philadelphia, PA: ISI Press, 1986).
EXAMPLE
Jan Hendrik Schön
http://archive.salon.com/tech/feature/2002/09/16/physics/
-
Science
Nature
http://physicsweb.org/article/news/6/9/15/
http://agenda.cern.ch/fullAgenda.php?ida=a01193
RECOMMENDATIONSWorkshop on the Open Archives Initiative (OAI)
and Peer Review Journals in Europe, CERN, Geneva Switzerland, March 22-24, 2001
“The participants were unanimous in their belief that the certification of scholarly work remains a fundamental part
of a system for scholarly communication.”
“It was [also] generally believed that the electronic environment allows for novel approaches to accord a
stamp of quality to scholarly works.”
Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1-8.
http://www.arl.org/sparc/SPB21_OAI.pdf
“Let us be imaginative in exploring the remarkable possibilities of this
brave new medium.”
Stevan Harnad, “Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals, in Scholarly Publication: The Electronic
Frontier, edited by Robin P. Peek and Gregory B. Newby (Cambridge MA: MIT Press, 1996).
http://www.ecs.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html
“Let us be more imaginative in exploring the remarkable possibilities of this
brave new medium.” With Apologies to Stevan Harnad
http://lockss.stanford.edu/
TM
LOCKSSFor centuries libraries and publishers have had stable roles: publishers produced information; libraries kept it safe for reader access. There is no fundamental reason for the online environment to force institutions to abandon these roles.
The LOCKSS model capitalizes on the traditional roles of libraries and publishers. LOCKSS creates low-cost, persistent digital "caches" of authoritative versions of http-delivered content.
LOCKSSThe LOCKSS software enables institutions to locally collect, store, preserve, and archive authorized content thus safeguarding their community's access to that content.
The LOCKSS model enforces the publisher's access control systems and, for many publishers, does no harm to their business models.
LAMPSS
Lots of Alternative Models Provide Sensible Solutions
DISCLAIMER
The alternative peer review models profiled are for informational and
educational purposes only and do not
necessarily constitute an endorsement.
ON
ALTERNATIVE PEER REVIEWNeo-ClassicalCertification-basedOpen Peer ReviewCommentary-basedCollaborately-filtered
Institution-basedCitation-basedIndex-basedMetadata-basedComputer-assistedNO Peer ReviewModerator-basedTier-based
NEO-CLASSICAL PEER REVIEW
Neo-Classical Peer Review
Neo-Classical Peer Review
Neo-Classical Peer Review
CERTIFICATION-BASED
Certification-Based“The process of pre-publication peer
review could be improved and become a more reliable indicator of manuscript quality if reviewers were trained in, and subsequently applied systematically, critical skills and use of a hierarchy of evidence to classify submitted articles being reviewed.”
Stephen Pritchard , “Peer Review - a Proposal for Change,”Paper presented at Thinking Globally - Acting Locally:
Medical Libraries at the Turn of an Era,8th European Conference of Health and Medical Libraries,
September 16-21, 2002, Deutsche Zentralbibliothek für Medizin, Köln, Germany.http://www.zbmed.de/eahil2002/abstracts/pritchard.pdf
OPEN PEER REVIEW
Open Peer ReviewIDENTIFICATION OF REVIEWERS / SIGNED REVIEWS
BMJ bmj.com
BioMed Centralbiomedcentral.com
eMJA (Medical Journal of Australia)www.mja.com.au/public/information/project.html
COMMENTARY-BASED
Commentary-basedReaders can comment before and/or after classic peer
review, or instead of classic peer review
Electronic Transactions on Artificial Intelligence (www.etaij.org)
OPEN REVIEW / REFEERING
Journal of Interactive Media in Education (www-jime.open.ac.uk)
PRE- AND POST- COMMENTARY
Psycoloquy
(psycprints.ecs.soton.ac.uk)
POST PEER REVIEW COMMENTARY)
Electronic Transactions on Artificial Intelligence
Journal of Interactive Media in Education
http://psycprints.ecs.soton.ac.uk/
COLLABORATIVELY-FILTERED
Collaboratively-Filtered
“DEFINITION: “Guiding people's choices of what to read, what to look at, what to watch, what to listen to (the filtering part); and doing that guidance based on information gathered from some other people (the collaborative part)."
Paul Resnick
http://www.cni.org/Hforums/cni-announce/1996/0031.html
ResearchIndex / CiteSeer
http://www.researchindex.com
INSTITUTION-BASED
Institution-BasedINSTITUTIONAL INITIATIVES
DSpace™ (MIT)www.dspace.org
eScholarship (University of California)escholarship.cdlib.org
Glasgow ePrint Service (University of Glasgow)eprints.lib.gla.ac.uk
CITATION-BASED
Citation-Based Citations to Open Access / OAI-compliant
documents are indicators of document importance
http://citebase.eprints.org/
INDEX-BASED
Index-Based INDEXING OF EPRINTS BY COMMERICAL ABSTRACTING AND
INDEXING SERVICE
Chemical Abstracts (American Chemical Society) (CAS) indexes select appropriate e-prints from the arXiv.org eprint server as well from the Chemical Preprint Service (Elsevier)Its “selection criteria for this kind of electronic document are essentially the same as for the traditional printed documents: they must report new information of chemical or chemistry-related interest and must be original publications. Also, the electronic publication must be publicly available and have some relative permanence ….”
Eric Shively / Chemical Abstracts Service
Index-Based “CAB Abstracts doesn’t currently include Eprints or Preprints, but we are looking at the implications and possible mechanisms for accessing and indexing Eprints and/or Preprints related to the applied life sciences.”
Tracy Shaw / CAB International
METADATA-BASED
Metadata-Based<oai-quality>
<category>internal</category>
<process>
peer review
</process>
<organization>
CERN
</organization>
<policies>
http://www.cern.ch/policies/review.html
</policies>
</oai-quality>
William Y. Arms, “Quality Control in Scholarly Publishing On The Web. What Are the Alternatives to Peer Review?” PowerPoint presentation given at the Workshop on the Open
Archives Initiatives (OAI) and Peer Review Journals in Europe, March 22-24, 2003, CERN, Geneva, Switzerland.http://www.cs.cornell.edu/wya/papers/CERN-2001.ppt
COMPUTER-ASSISTED
Computer-Assisted (1)SOFTWARE THAT ASSISTS IN THE EVALUATION OF A
SUBMITTED MANUSCRIPTA Software Program to Aid in Peer ReviewAlvar Loria and Gladys Faba
Objective: To characterize a personal computer-based software program developed as an aid to peer review of medical papers. Design: The software is a Windows-based application that records automatically a numeric score to a series of questions related to 8 sections of scientific papers (introduction, methods, results, and discussion, plus 4 other sections). The questions and sections vary according to type of paper (original reports, case reports, or reviews), and the final output is a score with a maximum of 100 for a "perfect" paper. The software was tested using a single reviewer to judge 289 papers (169 original reports, 50 case reports, and 70 reviews) from 44 Mexican medical journals. All statistical analysis of scores were done with nonparametric tests.
Computer-Assisted (2)Results: The paper scores ranged from 29 to 97 with slightly higher median and less dispersion of scores for reviews as compared with original reports and case reports, but these differences did not reach significance. Two observations suggest that the software operated reasonably well: a) there were some differences in the section scores by type of paper that agreed well with differences in their complexity; b) the journal scores showed an association with their number of original papers and their percentage of original papers (Kruskal-Wallis test, P=.06 and 0.07, respectively). Conclusions: The software operated reasonably well when used to compare the relative quality of 289 papers. The validity of the program is restricted in this study to the experience of 1 reviewer. An analysis of the raw scores helped in detecting some ambiguous and redundant questions that have been modified in an improved version. The program has a potential as a training tool for inexperienced reviewers or as a scorekeeper for experienced peer reviewers.
Alvar Loria and Gladys Faba, “A Software Program to Aid in Peer Review,” Abstract of paper presented at the Third International Congress on Biomedical Peer Review and Global Communications ,
September 18-20, 1997, Prague, Czech Republic. http://www.ama-assn.org/public/peer/arev.htm
NO PEER REVIEW
NO Peer Review
http://xxx.arXiv.cornell.edu
MODERATOR-BASED
Moderator-based (1)The intent of this model is to allow the widest range of scientific manuscripts to be archived, searched, and distributed electronically at the lowest possible cost.This would be accomplished through very minimal filtering and subsequent placement of eprints on a non-commercial archival server by a subject-specific Moderator appointed by a society (or consortia of societies).A society-appointed Editorial Board (with double-blind peer review approved by the non-profit Peer Review Inc. organization) would then the identify the most important materials from among these archived items, and the stamp of approval for these items would be included in a secondary Virtual Collection.
Moderator-based (2)There are no direct submissions to the Editorial Board; manuscripts would be directed to the Editorial Board in one of three ways:
1. nominated by the eprint Moderator upon receipt for the archival server,
2. notification sent to the Editorial Board when a threshold number of hits are generated by any one manuscript on the archive server, and
3. nominated by readers of material from the archive; this process requires a letter of support outlining the importance of the work to the Editorial Board.
Moderator-based (3)The Virtual Collection could be produced as a variety of products:
enhanced abstracts email threads (with comments) virtual reviews of sub-disciplines SDIs (selective dissemination of information) current awareness tools
This process: reduces the load on the Editorial Boards, which results in a faster review process; differentiates those items worthy of higher recognition from those worthy of archiving, making it easier for a reader to filter material, based upon a society and discipline authority (rather than commercial reasoning);provides for search/browse/sdi from the Virtual Collection for filtered info, reducing this more expensive option for only those items recognized as of the highest quality.
David Stern, “The eprint Moderator Model,” Newsletter on Serials Pricing Issues no. 214 (February 8, 1999).
http://www.lib.unc.edu/prices/1999/PRIC214.HTML#214.5
1
2 3
TIER-BASED
Tier-basedTwo separate domainsStandard Tier Any and all submissions would be accepted after a cursory examination of or other pro forma certification. The review process could be “minimally labor-intensive, perhaps relying primarily on an automated check of author institutional affiliation, prior publication record, research grant status, or other related background; and involve human labor primarily to adjudicate incomplete or ambiguous results of an automated pass.”
Tier-basedUpper Tier “At some later point (which could vary from article to article, perhaps with no time limit), a much smaller set of articles would be selected for the full peer review process. The initial selection criteria for this smaller set could be any of a variety of impact measures, to be determined, and based explicitly on their prior widespread and systematic availability and citability: e.g., reader nomination or rating, citation impact, usage statistics, editorial selection, ... .”
Paul Ginsparg, “Can Peer Review be Better Focused?,” Science & Technology Libraries 22 No. 3/4 (In press).
http://arxiv.org/blurb/pg02pr.html
DISCLAIMER
Presented for Your Consideration
OFF
FREEDOM OF IDEAS
http://www.nrm.org/exhibits/current/four-freedoms.html
‘FUTURE OF IDEAS’ The explosion of innovation we have seen
in the environment of the Internet was not conjured from some new, previously
unimagined technological magic; instead, it came from an ideal as old as the nation. Creativity flourished there because the
Internet protected an innovation commons.
‘FUTURE OF IDEAS’
The Internet’s very design built a neutral platform upon which the widest range of
creators could experiment.
The legal architecture surrounding it protected this free space so that culture and information –
the ideas of our era–could flow freely and inspire an unprecedented breadth of expression.
Lawrence Lessig, The Future of Ideas: The Fate of the Commons in a Connected World
. (New York: Random House, c2001).
“IT’S
NOT
ABOUT
PUBLICATION;
IT’S
ABOUT
IDEAS.”
The Association of Learned and Professional Society Publishers (ALPSP) Survey
Authors and Electronic PublishingScholarly research communication has seen far-reaching developments in recent years. Most journals are now available online as well as in print, and numerous electronic-only journals have been launched; the Internet opens up new ways for journals to operate. Authors have also become conscious of alternative ways to communicate their findings, and much has been written about what they ought to think.
ALPSP felt that it would be timely to discover what they actually thought and what they actually did. This survey aimed to discover the views of academics, both as authors and as readers. Some 14,000 scholars were contacted across all disciplines and all parts of the world, and nearly 9% responded; their detailed comments make thought-provoking reading.
Alma Swan and Sheridan Brown. Authors and Electronic Publishing: The ALPSP Research Study on Authors' and Readers’ Views of Electronic Research Communication. (West Sussex, UK: The Association of Learned
and Professional Society Publishers, 2002).http://www.alpsp.org/pub5.htm
“When asked to predict what would be the most common form of quality
control in five years time, only a bare majority answered
‘traditional peer review’.”
Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): 247-258.
Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf
FURTHERMORE …16% said that the referees would no longer be anonymous27% said that traditional peer review would be supplemented by post-publication commentary 45% expected to see some changes in the peer-review system within the next five years
Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): 247-258.
Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf
Importance of the Peer Review Process
0102030405060708090
100Peer-reviewed
Refs' commentspublished
Referees identified
Public commentary oneprints
Post-publication publiccommentary
Ability to submitcomments
http://www.alpsp.org/pub5.ppt
What is Gray/Grey Literature ?Papers are often written to inform funding bodies
about the results of research projects, to support grant applications, to inform rapidly a specific scientific community, to present preliminary results at conferences or as dissertations.
Such material is disseminated quickly, often in limited numbers, before or without the formal publication process. Such documents are called non-conventional or grey literature.
http://www.kb.nl/infolev/eagle/what_is_gl.htm
The Value of Grey LiteratureGrey literature is really a type of informal
communication, which on a scale of formality, fits in somewhere between conversation and
normal publication. A formal publication may follow later but in many cases - contrary to the
common assumption - these papers may not been made publicly available at all.
http://www.kb.nl/infolev/eagle/what_is_gl.htm
Nevertheless, grey publications may contain comprehensive, concrete and up-to-date information on research findings, and investigations have shown, that even when grey documents are published officially at a
later stage, detailed information on techniques, methods, measured values and details of experiments are
frequently omitted.
For these details of importance for further research, the non-conventional literature is then the first and only
source of information.
http://www.kb.nl/infolev/eagle/what_is_gl.htm
Veterinary Medicine12 Major Veterinary Medicine JournalsOverall, 6.38% of cited literature was Gray/Grey Literature
The figures for individual journals ranged from about 2.5 % to 10% gray/grey literature
Research journals cited a higher percentage of Gray/Grey Literature than did Clinical titles
William H. Weise and Nancy Pelzer, “Bibliometric Study of Grey Literature in Core Veterinary Medicine Journals,” Journal of the
Medical Library Association 91 no. 4 (October 2003): In press.
Indexing and Abstracting Services
SIGLE: System for Information on Grey Literature
National Technical Information Service (NTIS)
PsychINFO (Psychological Abstracts)
SIGLESystem for Information on Grey
LiteratureGrey literature documents covered
by SIGLE are technical or research reports, preprints, committee reports, working papers, dissertations, conference papers, discussion and policy papers, government reports, market surveys, etc.
http://www.kb.nl/infolev/eagle/frames.htm
SIGLESystem for Information on Grey
LiteratureNo. of Records | Category
4,158 | Aeronautics
17,044 | Agriculture, plant & veterinary sciences
17,668 | Environmental pollution, protection & control
256,657 | Humanities, psychology & social sciences
81,269 | Biological & medical sciences
25,089 | Chemistry
http://www.kb.nl/infolev/eagle/frames.htm
NTIS (National Technical Information Service)
NTIS Database provides bibliographic data and abstracts of unclassified and publicly
available information from research reports, journal articles, data files, computer programs and audio visual products, from U.S. and non-
U.S. governmental, organizational, and commercial sources
The NTIS Database produced by the National Technical Information Service, is the preeminent resource for
accessing the latest U.S. government-sponsored research and worldwide scientific, technical, engineering, and
business-related information.
http://www.csa2.com/csa/factsheets/ntis.shtml
Subject CoverageAdministration and Management
Aeronautics & Aerodynamics
Agriculture Behavior & Society
Business Chemistry Communications Computer Science
Education Energy Engineering Environmental Sciences
Health Care International Trade
Library & Information Science
Materials Sciences
Mathematical Sciences
Natural Resources & Earth Sciences
Nuclear Science Physics
Regulations Technology Tele-communications
Transportation
PsycINFOPsycINFO provides access to international literature in
psychology and related disciplines. Unrivaled in its depth of psychological coverage and respected worldwide for its high
quality, the database is enriched with literature from an array of disciplines related to psychology such as psychiatry, education, business, medicine, nursing, pharmacology, law, linguistics, and
social work.
http://www.csa2.com/csa/factsheets/psycinfo.shtml
PsycINFO includes psychological research and its applications; the database is of prime relevance to many industries and research
establishments worldwide. The sources include over 1,400 professional journals, chapters, books, reports, theses and
dissertations, published internationally.
Subject CoverageApplied Psychology
Communication Systems
Developmental Psychology
Educational Psychology
Experimental
Psychology
Personality Psychological and physical disorders
Professional personnel issues
Physiological Psychology and
Neuroscience
Psychometrics
And Statistics
Social Psychology
Treatment and Prevention
Database Coverage Size
SIGLE 1976 –
Present
781,410 records(November 2002)
NTIS 1964 - Present 2,168,400
records(October 2001)
PsycINFO 1872-
Present
1,870,180
records(September 2002)
DATABASE COVERAGE AND SIZE
http://www.neci.nec.com/~lawrence/papers/online-nature01/
Conference papers are typical gray/grey literature!
EPrints are Gray/Grey Literature
Daniela Luzi (1998) “E-Print Archives: a New Communication Pattern for Grey Literature,”
Interlending & Document Supply 26 no. 3 (1998): 130-139.
Gray/Grey Literature“It’s good enough,
it’s smart enough,
and
doggone it, people use it!”
With apologies to Stuart Smalley
http://www.openarchives.org/Register/BrowseSites.pl
http://software.eprints.org/
June 2003
http://eprints.anu.edu.au/
http://caltechcstr.library.caltech.edu/
http://eprints.lub.lu.se/
http://ndltdpapers.dlib.vt.edu:9090/
NETWORKED DIGITAL LIBRARY OF THESES AND DISSERTATIONS
http://dspace.org/index.html
http://rocky.dlib.vt.edu/~etdunion/cgi-bin/browse.pl
http://www.ncstrl.org/
Quality Control in Scholarly Publishing on the Web
"Most of the high quality materials on the Web are not peer-reviewed and much
of the peer-reviewed literature is of dubious quality.”
William Y. Arms, "What Are the Alternatives to Peer Review? Quality Control in Scholarly Publishing On The Web."
Journal of Electronic Publishing, 8 no. 1 (August 2002).
http://www.press.umich.edu/jep/08-01/arms.html
http://www.update-software.com/Cochrane/MR000016.pdf
Cochrane Methodology ReviewDespite its widespread use and costs, little hard evidence exists that peer review improves the quality of published biomedical research.
There had never even been any consensus on its aims and that it would be more appropriate to refer to it as ‘competitive
review’.Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”
BMJ 326 (February 1, 2003): 241http://bmj.com/cgi/reprint/326/7383/241/a.pdf
Cochrane Methodology ReviewOn the basis of the current evidence, ‘the practice of peer review is based on faith in its effects, rather than on facts,' state the authors, who call for large, government funded research programmes to test the effectiveness of the [classic peer review] system and investigate possible alternatives.
Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”BMJ 326 (February 1, 2003): 241
http://bmj.com/cgi/reprint/326/7383/241/a.pdf
Cochrane Methodology ReviewThe use of peer-review is usually assumed to raise the quality of the end-product (i.e. the journal or scientific meeting) and to provide a mechanism for rational, fair and objective decision-making. However, these assumptions have rarely been tested.
Tom O. Jefferson, Phil Alderson, Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of
Biomedical Studies. (Middle Way, Oxford:Update Software Ltd, 2003).
http://www.update-software.com/Cochrane/MR000016.pdf
Cochrane Methodology ReviewThe available research has not clearly identified or assessed the impact of peer-review on the more important outcomes (importance, usefulness, relevance, and quality of published reports)
… [G]iven the widespread use of peer-review and its importance, it is surprising that so little is known of its effects
Tom O. Jefferson, Phil Alderson,Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of
Biomedical Studies. (Middle Way, Oxford:Update Software Ltd, 2003).
http://www.update-software.com/Cochrane/MR000016.pdf
http://www.biomedcentral.com/news/20030203/04/
Royal Society
UNCITEDNESS
David P. Hamilton, "Publishing by and for? -- the numbers,” Science (New Series) 250 (4986) (December 7 1990): 1331-1332.http://www.garfield.library.upenn.edu/papers/hamilton1.html
David P. Hamilton,“Research Papers: Who’s Uncited Now?,” Science (New Series) 251 (4989) (January 4, 1991): 25
http://garfield.library.upenn.edu/papers/hamilton2.html
David P. Hamilton,“Research Papers: Who’s Uncited Now?,” Science (New Series) 251 (4989) (January 4, 1991): 25
http://garfield.library.upenn.edu/papers/hamilton2.html
REJECTED CITATION CLASSICSNOBEL PRIZE RESEARCH
Severo Ochoa
Polynucleotide phosphorylase
Hans Krebs
Citric acid cycle
Rosalind Yalow
Radioimmunoassay
Harmut Michel
Photosynthetic processesJuan Miguel Campanario, “Commentary: On Influential Books and Journal Articles
Initially Rejected Because of Negative Referees’ Evaluations, Science Communication 16 no. 3 (March 1995): 306-325
PEER REVIEW: PURPOSESPeer review helps to ensure that published research is:
Important ? Original ?
Timely ? Technically-reliable ?
Internally consistent ? Well-presented ?
Benefited from guidance by experts ?
Carin M. Olson, “Peer Review of the Biomedical Literature,” American Journal of Emergency Medicine 8 no.4 (July 1990): 356-368.
FILTERING (1)UpStream / DownStream
“Researchers look at … [certain types] of electronic publications because, despite being tentative, may be relevant to their
work. Researchers are expected to do their own ‘downstream-filtering’ of relevant
information, which in the electronic world can be facilitated by providing meta-
information.”
FILTERING (2)UpStream / DownStream
Some have expressed the concern that having non-peer reviewed documents with peer-
reviewed documents on the same server would ‘contaminate’ the latter and compromise its
quality: Readers could have trouble in distinguishing
the different sections ‘Making non-peer-reviewed as well as peer-
reviewed material will confuse both scientists and the public … .’
FILTERING (3) UpStream / DownStream
“‘However, this perhaps belittles the ability of scientists to recognize different levels of evidence and to be able to interpret [quality] labels that could make clear that certain materials is non-peer-reviewed content’ after all this is
the age of transparency rather than paternalism … .”
Gunther Essenbach, “The Impact of Preprint Servers and Electronic Publishing on Biomedical Research,”
Current Opinion in Immunology 12 no. 5 (October 2000): 499-503.
http://yi.com/home/EysenbachGunther/scans/Eysenbach2000e_CurrOpImmunol_preprint_servers.pdf
INVISIBLE HAND OF CLASSICAL PEER REVIEW
“The refereed journal literature needs to be freed from both paper and its costs, but not from peer review, whose ‘invisible hand’ is what maintains its quality.”
Stevan Harnad
http://www.presidentmoron.com
INVISIBLE HAND OF CLASSICAL PEER REVIEW
INVISIBLE HAND OF CLASSICAL PEER REVIEW
“Human nature being what it is, it cannot be altogether relied upon to police itself. Individual exceptions there may be, but to treat them as the
rule would be to underestimate the degree to which our potential unruliness is vetted by
collective constraints, implemented formally.”
Stevan Harnad, “The Invisible Hand of Peer Review,” Exploit Interactive no. 5 (April 2000).
http://www.exploit-lib.org/issue5/peer-review/
INVISIBLE HAND OF CLASSICAL PEER REVIEW
“The system is not perfect, but it is what has vouchsafed us our refereed journal literature to
date, such as it is, and so far no one has demonstrated any viable alternative to having experts judge the work of their peers, let alone
one that is at least as effective in maintaining the quality of the literature as the present imperfect
one is.”
INVISIBLE HAND OF CLASSICAL PEER REVIEW
“Remove that invisible constraint -- let the authors be answerable to no one but the general users of the
Archive [arXiv.org] (or even its self-appointed "commentators") -- and watch human nature take its
natural course, standards eroding as the Archive devolves toward the canonical state of unconstrained
postings: the free-for-all chat-groups of Usenet … , that Global Graffiti Board for Trivial Pursuit -- until
someone re-invents peer review and quality control.”
Stevan Harnad, “The Invisible Hand of Peer Review,” Exploit Interactive no. 5 (April 2000).
http://www.exploit-lib.org/issue5/peer-review/
INVISIBLE HANDS
INVISIBLE HANDS
Personal reputation
Institutional reputation
Pride
Self-respect
Professional respect
Peer pressure
‘Critical Peer Response’
Invisible College
Self-Archiving-Process-Itself
Open access
Common Sense
Self-correcting dynamics
RECOMMENDATIONSWorkshop on the Open Archives Initiative (OAI)
and Peer Review Journals in Europe, CERN, Geneva Switzerland,March 22-24, 2001
“It was [also] generally believed that the electronic environment allows for novel
approaches to accord a stamp of quality to scholarly works.”
Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful
Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1-8.
http://www.arl.org/sparc/SPB21_OAI.pdf
Examples of new metrics that can be extracted from a fully electronic communication system are:
Usage counts of a work
Automatically extracted citation information with a scope beyond the ISI- core journals
Amount of discussion generated by a paper submitted in a system with open peer review and peer comment
Etc.Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful
Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1-8.
http://www.arl.org/sparc/SPB21_OAI.pdf
Scientific Publishing as RhetoricThe problems with peer review become evident once the fact that science has a rhetorical element is accepted. On the one hand, the traditional mode of peer review obscures the problems of reference and the rhetorical dimension of science. The rhetorical process which is at the heart of science and peer review conveniently disappears with the final publication of the manuscript. In its place is an ideal typical representation (the scientific paper) of the realist assumptions about empirical reference. All the academic world sees is a polished manuscript where the personal involvement of the researcher and reviewers has been systematically eliminated.
Mike Sosteric, “Interactive Peer Review: A Research Note,” Electronic Journal of Sociology 2 no. 1 (1996).
http://socserv.socsci.mcmaster.ca/EJS/vol002.001/SostericNote.vol002.001.html
‘IDEAL SPEECH SITUATION’
A theoretical construct that describes the ideal type of interpersonal interaction that should exist in a rhetorical situation.
Jürgen Habermas
IDEAL SPEECH SITUATIONthe ideal speech situation permits each interlocutor an
equal opportunity to initiate speech;there is mutual understanding between interlocutors;there is space for clarification; all interlocutors are equally free to use of any speech
act; there is equal power over the exchange.
Applied in the context of peer, the Ideal Speech Situation ‘would permit unimpeded authorial initiative, endless rounds of give and take, [and] unchecked openness among authors, editors, and referees.’
Mike Sosteric, “ Interactive Peer Review: A Research Note,” Electronic Journal of Sociology 2 no. 1 (1996).
http://socserv.socsci.mcmaster.ca/EJS/vol002.001/SostericNote.vol002.001.html
CONTINUAContinuum of PUBLICATION
(‘Scholarly Skywriting’)
WEAK
MEDIUM
Continuum of REVIEW (‘Scholarly Skyreading’)
STRONGSTRONG
OPEN ACCESS and
OPEN RETRIEVAL without
OPEN USEIncongruent
Contradictory Ironic
Paradoxical
Cognitively Dissonant
Three-Legged Stool
ACCESS -- RETRIEVAL -- USE
ACCESS
OPEN ACCESS INITIATIVES
RETRIEVAL
OPEN ARCHIVES INITIATIVE FOR METADATA HARVESTING
USE
OPEN SCHOLARSHIP
ACCESS -- RETRIEVAL -- USEACCESS -- RETRIEVAL -- USE
Lots of Alternative ModelsProvide Sensible Solutions
Four-Legged Stool
ACCESS -- RETRIEVAL -- USE -- NAVIGATION
INFORMATION OVERLOAD
OAIster
A search engine for freely available, difficult-to-access, academically-oriented digital resources that are OAI -compliant
http://oaister.umdl.umich.edu/
University of Michigan Digital Library Production Service
institutional repositories
departmental repositories
e-Journal collections
technical reports
dissertations and theses
discipline eprint collections
working papers
Internet resources
audio
video
images
Cognitive Psychology
Open Archives Initiative Protocol for Metadata Harvesting
(OAI-PMH)
METADATA ELEMENT for QUALITY
ADD
“Within the framework of OAI,
there is a need for a new protocol for certification.
There was strong support
for the extension of the usage of the
OAI protocol beyond discovery-related
metadata. Given the focus of the [1st OAI]
workshop on peer review, concrete
actions were suggested to address the
exchange of certification-related
metadata using the OAI protocol in a
trusted environment.”
http://www.arl.org/sparc/SPB21_OAI.pdf
Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful Owners,” European
Science Foundation Policy Briefing 21 (April 2003): 1-8.
QUALITY METADATA (1)<oai-quality>
<category>internal</category>
<process>
peer review
</process>
<organization>
CERN
</organization>
<policies>
http://www.cern.ch/policies/review.html
</policies>
</oai-quality>
William Y. Arms, “Quality Control in Scholarly Publishing On The Web. What Are the Alternatives to Peer Review?” PowerPoint presentation given at Workshop on the Open Archives
Initiatives (OAI) and Peer Review Journals in Europe, March 22-24, 2003, CERN, Geneva, Switzerlandhttp://www.cs.cornell.edu/wya/papers/CERN-2001.ppt
QUALITY METADATA (2)<oai-quality>
<category>internal</category>
<rating>
*****
</rating>
<organization>
42
</organization>
<policies>
http://www.cern.ch/policies/review.html
</policies>
</oai-quality>
CERTIFICATION SERVICESFaculty of 1,000,000
CERTICATION SERVICESNew roles for Indexing and Abstracting Services
Expanded Role for Learned and Professional Societies
Establishment of Formal/Commercial Reviewing Services
PEER REVIEW: STRENGTHSThe underlying strength of peer review is“…the concerted effort by large numbers of researchers and scholars who work to assure that valid and valuable works are published and conversely to assure that invalid or non-valuable works are not published … .”
Anne C. Weller, Editorial Peer Review: Its Strengths and Weaknesses. (Medford, NJ: Information Today, 2001).
Faculty of 1000 / BioMed CentralBioMed Central (biomedcentral.com) publishes Faculty of 1000 (F1000), the leading literature evaluation service and “new online research tool that highlights the most interesting papers in biology, based on the recommendations of over 1000 leading scientists.” F1000 is managed “by scientists for scientists” …. [and] provide[s] a rapidly updated consensus map of the important papers and trends across biology.”
www.facultyof1000.com
Faculty of 1000 / BioMed CentralAmong its many benefits, F1000:systematically organizes and evaluates the mass of information within scientific literature; provides scientists with a continuously updated insider's guide to the most important papers within any given field of research;highlights papers on the basis of their scientific merit rather than the journal in which they appear;offers the researcher a consensus of recommendations from well over 1000 leading scientists; and,offers an immediate rating of individual papers by the authors' peers, and an important complement to the indirect assessment provided by the journal impact factor.
Faculty of 1000 / BioMed CentralWithin the F1000, the entire field of biology
is divided into 16 subject areas (‘Faculties’)(e.g., ‘Biochemistry,’, Cell Biology, ‘Microbiology’). Each ‘Faculty’ is subdivided into three (3) to twelve (12) ‘Sections,’ (e.g., Biochemistry: Biocatalysis, Molecular evolution, Protein folding), with each section comprised of between 10 to 50 faculty members. F1000 seeks to invite the best internationally known scientists in each represented field and to involve both experienced and younger investigators.
“Peer review is a quality-control and certification (QC/C) filter necessitated by the vast scale of learned research
today. Without it, no one would know where to start reading in the welter of new work reported every day, nor
what was worth reading, and believing, and trying to build one’s own further research upon.”
Stevan Harnad, “Free at Last: The Future of Peer-Reviewed Journals,”
D-Lib Magazine 5 no. 12 (December 1999)
http://www.dlib.org/dlib/december99/12contents.html
SEIZE THE E!
Embrace the potential of the digital environment to facilitate access, retrieval,
use, and navigation of electronic scholarship.
OPEN NAVIGATION
New Age Navigation:Innovative Interfaces for
Electronic Journals
Gerry McKiernan
The Serials LibrarianFall 2003
SUMMARY. While it is typical for electronic journals to offer conventional search features similar to those
provided by electronic databases, a select number of e-journals have also made available higher-level access options as well. In this article, we review several novel
technologies and implementations that creatively exploit the inherent potential of the digital environment to
further facilitate use of e-collections.
Gerry McKiernan, “New Age Navigation: Innovative Interfaces for Electronic Journals,”
The Serials Librarian, Fall 2003.
http://www.coleonline.us/serialslibrarian/
http://www.highwire.org
Topic Map
Topic Map
Topic Map
Topic Map
Topic Map
Secondary Screen
Topic Map
Topic Map
http://www.inxight.com
TopicMap is based on the Hyperbolic Tree SDK for Java, licensed from Inxight
Software, Inc., a spin-off company from Xerox PARC, and leading provider of
Unstructured Data Management solutions for accessing, analyzing and delivering
information.
Semio
Automated categorization software technologyhttp://www.entrieva.com/
Kohonen Self-Organizing Maps
http://websom.hut.fi/websom/milliondemo/html/root.html
Kohonen Self-Organizing Maps
http://websom.hut.fi/websom/milliondemo/html/1_cx5.html
Kohonen Self-Organizing Maps
http://websom.hut.fi/websom/milliondemo/html/2_gx10.html
Kohonen Self-Organizing Maps
http://websom.hut.fi/websom/milliondemo/getnd.cgi?32323
Kohonen Self-Organizing Maps
http://websom.hut.fi/websom/cgi-bin/getfile.cgi/comp.ai/39340
http://www.springer.de/books/toc/3540679219-c.pdf
“There are some excuses, but at the bottom it will be seen to be the
sluggishness of human nature and its superstitious cleavage to old habits.”
Stevan Harnad, “Free at Last: The Future of Peer-Reviewed Journals,”
D-Lib Magazine 5 no. 12 (December 1999)
http://www.dlib.org/dlib/december99/12contents.html
“… [In] the digital world, the evaluation process stands ready to be reinvented in a clear, rational
way by the relevant research communities themselves.”
Jean-Claude Guédon,
In Oldenburg’s Long Shadow: Librarians, Research Scientists, Publishers,
and the Control of Scientific Publishing.
(Washington, D.C.: Association of Research Libraries, •2001), 54.
http://www.arl.org/arl/proceedings/138/guedon.html
</ENDQUOTE>
“The Medium is the Message …
And
the Method.”
With apologies to Marshall McLuhen
THANK YOU FOR YOUR ATTENTION!
Alternative Peer Review:Quality Management
for 21st Century Scholarship
Gerry McKiernanScience and Technology Librarian
and Bibliographer
Iowa State University Library
Ames IA
OPEN MIND
REVISED VERSION 1.01
July 27, 2003
11:30 AM