Download - UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley
‘Peer review is dead: long live peer review!’
Michael WillisWiley, Oxford, UK
[email protected] @ctyerkes
1
A venerable legacy
2
"Philosophical Transactions Volume 1 frontispiece" by Henry Oldenburg - Philosophical Transactions. Licensed under CC BY 4.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:Philosophical_Transactions_Volume_1_frontispiece.jpg
Augustus Bozzi Granville" by Unknown - Wellcome Images. Licensed under Public Domain via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:Augustus_Bozzi_Granville.jpg#mediaviewer/File:Augustus_Bozzi_Granville.jpg
A venerable tradition
3
• Augustus Bozzi Granville (1783-1872)
• First medical autopsy on an Ancient Egyptian Mummy, described to the Royal Society in 1825
• Critique of peer review as conducted by Royal Society
Science without a head; or, the Royal Society dissected
London, 1830
The Royal Society in the 19th century; being a statistical summary of its labours during the last thirty-five years, with many original tables and official documents (never before published)
London, 1836
Polemic against the old order
4
Peer review is…
Arbitrary
‘Much oftener is the fate of a paper committed to the chances of the mere yea-and-nay box, than to the decision of a competent judge instructed to offer a preliminary opinion upon its merits.’
Incompetent
‘At many of these meetings, members of the committee of papers have been present who have not the smallest pretension to any knowledge whatever of the subject under consideration.’
Critique of method
5
‘It has been said that the warp that holds the complex fabric of science together is peer review, and the woof is the noise made by scientists who complain about it.’
Jukes, TH. Peer review. Nature 265 (1977); 203
‘Those who complain of unfair treatment at the hands of reviewers would do well to remember that 400 years ago they would have had—at best—the choice between the burning of their manuscripts or of themselves.’
Grivell, L. Through a glass darkly: The present and the future of editorial peer review. EMBO reports 7 (2006); 567-570
Leaping forward…
6
‘…slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless for detecting fraud…’
Smith, R. Peer review: reform or revolution? BMJ 315 (1997): 759–760
A damning indictment
7
Nails in the coffin of peer review?
8
It’s a black box
It’s a black box
There are too few reviewers to do the work
There are too few reviewers to do the work
It takes too longIt takes too long
It costs too muchIt costs
too much
Microsoft Clip ArtIt’s not rigorous enough
It’s not rigorous enough
The web changes
everything
The web changes
everything
How strong are the nails?
9
Why does this matter?
furthering of scientific
endeavour
health and welfare of society
public and academic
confidence in scientific record
integrity of scientific record
10
‘Many forms of Government have been tried, and will be tried in this world of sin and woe. No-one pretends that democracy is perfect or all-wise. Indeed, it has been said that democracy is the worst form of Government except all those other forms that have been tried from time to time.’
Peer review resurgens
11
peer reviewpeer review
traditional peer review
traditional peer review
traditional peer review
traditional peer review
peer reviewpeer review
"Sir Winston S Churchill" by United Nations Information Office, New York - Library of Congress, Reproduction number LC-
USW33-019093-C© US Federal Government
Peer review: alive and kicking
12
clipartp
anda.com
‘…peer review is not only alive and kicking, but apparently increasing its influence, despite the many potential (or invented) threats posed by a rapidly unfolding and enveloping digital environment…’
Peer review: still king in the digital age, Nicholas et al., Learned Publishing 28(2015):15-21
Peer review under increased scrutiny
13
—Research indicates that the vast majority of researchers still value and trust peer review
—More scholars are looking at peer review as a scientific object of study
—There are now more organisations devoted to understanding and improving peer review than ever before
—There are several new products and services designed to preserve and enhance peer review
‘Four reasons to feel good about the future of peer review’
14
Eric Hallhttp://www.researchinformation.info/news/news_story.php?news_id=186210 March 2015
Peer review in scientific publications (July 2011)http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/856.pdf
‘…despite the many criticisms and the little solid evidence on its efficacy, editorial peer review is considered by many as important and not something that can be dispensed with.’
‘In order for current peer-review practices to be optimised and innovative approaches introduced, publishers, research funders and the users of research outputs (such as industry and government) must work together.’
‘…we encourage the prudent use of online tools for post-publication review and commentary as a means of supplementing pre-publication review.’
‘The integrity of the peer-review process can only ever be as robust as the integrity of the people involved.’
UK Parliament on peer review
15
Peer review survey, Sense About Science (2009)http://www.senseaboutscience.org/data/files/Peer_Review/Peer_Review_Survey_Final_3.pdf
—What do you think of peer review overall?
—What value does peer review add to research output?
—How can peer review be improved?
Researchers on peer review
16
Researchers on peer reviewWhat do you think of peer review overall?
17
Researchers on peer reviewWhat value does peer review add?
18
Researchers on peer reviewHow can peer review be improved?
19
Joynson C and Leyser O 2015 [v1; ref status: approved 1, http://f1000r.es/53j] F1000Research 2015, 4:66 (doi: 10.12688/f1000research.6163.1)
The culture of scientific research, Joynson C and Leyser O.Conclusions of 2014 survey by Nuffield Council on Bioethics
Researchers on peer reviewTraining: an opportunity for publishers
21
Find out what others are doing in your area of
research
Critique the research
undertaken by others
Review manuscripts in your area of research
Inform your own research
Conduct your own research
The virtuous circle of peer review and
research community engagement
Reviewer accountability 1:A creditable activity
23Spectrum (Official newsletter of the Australian Institute of Radiography), November 2014
Reviewer accountability 2:Public recognition
24
showcaseshowcase
get creditget credit
Reviewer accountability 3:The Hippocratic Oath of peer review
25
transparencytransparency reproducibilityreproducibility
Common themes for authors
26
transparencytransparency
rigourrigour
speedspeed
collaborationcollaboration
‘Aim high, publish fast’
27
Peer review variables
Transparency
Timing
Submission process
Assessment of…
Reviewerselection
Decision process
-Open peer review-Closed peer review
-Pre-publication-Post-publication
-Direct to a journal-To a 3rd party-Transfer within “family”
-Science only-Science + novelty or impact
-Independent review-Interactive review -Collaborative review
-Authors recruit reviewers-Editors recruit reviewers-Reviewers “bid” for papers
‘…publishers would welcome more guidance from key sections of the research community on the kinds of peer review services they want publishers to provide, and on the purposes that they should seek to fulfil.’
Scholarly Communication and Peer Review: The Current Landscape and Future Trends, Research Information Network
CIC, March 2015
Peer review evolution:A collaborative effort
29
The journey of peer review evolution
30
Listen to the community
Work with the community
Do more research
Be flexible
Train, educate, evaluate
Collaborate with stakeholders
Collaborate with stakeholders
31
32