Download - UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley
![Page 1: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/1.jpg)
‘Peer review is dead: long live peer review!’
Michael WillisWiley, Oxford, UK
[email protected] @ctyerkes
1
![Page 2: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/2.jpg)
A venerable legacy
2
"Philosophical Transactions Volume 1 frontispiece" by Henry Oldenburg - Philosophical Transactions. Licensed under CC BY 4.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:Philosophical_Transactions_Volume_1_frontispiece.jpg
![Page 3: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/3.jpg)
Augustus Bozzi Granville" by Unknown - Wellcome Images. Licensed under Public Domain via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:Augustus_Bozzi_Granville.jpg#mediaviewer/File:Augustus_Bozzi_Granville.jpg
A venerable tradition
3
• Augustus Bozzi Granville (1783-1872)
• First medical autopsy on an Ancient Egyptian Mummy, described to the Royal Society in 1825
• Critique of peer review as conducted by Royal Society
![Page 4: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/4.jpg)
Science without a head; or, the Royal Society dissected
London, 1830
The Royal Society in the 19th century; being a statistical summary of its labours during the last thirty-five years, with many original tables and official documents (never before published)
London, 1836
Polemic against the old order
4
![Page 5: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/5.jpg)
Peer review is…
Arbitrary
‘Much oftener is the fate of a paper committed to the chances of the mere yea-and-nay box, than to the decision of a competent judge instructed to offer a preliminary opinion upon its merits.’
Incompetent
‘At many of these meetings, members of the committee of papers have been present who have not the smallest pretension to any knowledge whatever of the subject under consideration.’
Critique of method
5
![Page 6: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/6.jpg)
‘It has been said that the warp that holds the complex fabric of science together is peer review, and the woof is the noise made by scientists who complain about it.’
Jukes, TH. Peer review. Nature 265 (1977); 203
‘Those who complain of unfair treatment at the hands of reviewers would do well to remember that 400 years ago they would have had—at best—the choice between the burning of their manuscripts or of themselves.’
Grivell, L. Through a glass darkly: The present and the future of editorial peer review. EMBO reports 7 (2006); 567-570
Leaping forward…
6
![Page 7: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/7.jpg)
‘…slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless for detecting fraud…’
Smith, R. Peer review: reform or revolution? BMJ 315 (1997): 759–760
A damning indictment
7
![Page 8: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/8.jpg)
Nails in the coffin of peer review?
8
It’s a black box
It’s a black box
There are too few reviewers to do the work
There are too few reviewers to do the work
It takes too longIt takes too long
It costs too muchIt costs
too much
Microsoft Clip ArtIt’s not rigorous enough
It’s not rigorous enough
The web changes
everything
The web changes
everything
![Page 9: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/9.jpg)
How strong are the nails?
9
![Page 10: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/10.jpg)
Why does this matter?
furthering of scientific
endeavour
health and welfare of society
public and academic
confidence in scientific record
integrity of scientific record
10
![Page 11: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/11.jpg)
‘Many forms of Government have been tried, and will be tried in this world of sin and woe. No-one pretends that democracy is perfect or all-wise. Indeed, it has been said that democracy is the worst form of Government except all those other forms that have been tried from time to time.’
Peer review resurgens
11
peer reviewpeer review
traditional peer review
traditional peer review
traditional peer review
traditional peer review
peer reviewpeer review
"Sir Winston S Churchill" by United Nations Information Office, New York - Library of Congress, Reproduction number LC-
USW33-019093-C© US Federal Government
![Page 12: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/12.jpg)
Peer review: alive and kicking
12
clipartp
anda.com
‘…peer review is not only alive and kicking, but apparently increasing its influence, despite the many potential (or invented) threats posed by a rapidly unfolding and enveloping digital environment…’
Peer review: still king in the digital age, Nicholas et al., Learned Publishing 28(2015):15-21
![Page 13: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/13.jpg)
Peer review under increased scrutiny
13
![Page 14: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/14.jpg)
—Research indicates that the vast majority of researchers still value and trust peer review
—More scholars are looking at peer review as a scientific object of study
—There are now more organisations devoted to understanding and improving peer review than ever before
—There are several new products and services designed to preserve and enhance peer review
‘Four reasons to feel good about the future of peer review’
14
Eric Hallhttp://www.researchinformation.info/news/news_story.php?news_id=186210 March 2015
![Page 15: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/15.jpg)
Peer review in scientific publications (July 2011)http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/856.pdf
‘…despite the many criticisms and the little solid evidence on its efficacy, editorial peer review is considered by many as important and not something that can be dispensed with.’
‘In order for current peer-review practices to be optimised and innovative approaches introduced, publishers, research funders and the users of research outputs (such as industry and government) must work together.’
‘…we encourage the prudent use of online tools for post-publication review and commentary as a means of supplementing pre-publication review.’
‘The integrity of the peer-review process can only ever be as robust as the integrity of the people involved.’
UK Parliament on peer review
15
![Page 16: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/16.jpg)
Peer review survey, Sense About Science (2009)http://www.senseaboutscience.org/data/files/Peer_Review/Peer_Review_Survey_Final_3.pdf
—What do you think of peer review overall?
—What value does peer review add to research output?
—How can peer review be improved?
Researchers on peer review
16
![Page 17: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/17.jpg)
Researchers on peer reviewWhat do you think of peer review overall?
17
![Page 18: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/18.jpg)
Researchers on peer reviewWhat value does peer review add?
18
![Page 19: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/19.jpg)
Researchers on peer reviewHow can peer review be improved?
19
![Page 20: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/20.jpg)
Joynson C and Leyser O 2015 [v1; ref status: approved 1, http://f1000r.es/53j] F1000Research 2015, 4:66 (doi: 10.12688/f1000research.6163.1)
The culture of scientific research, Joynson C and Leyser O.Conclusions of 2014 survey by Nuffield Council on Bioethics
![Page 21: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/21.jpg)
Researchers on peer reviewTraining: an opportunity for publishers
21
![Page 22: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/22.jpg)
Find out what others are doing in your area of
research
Critique the research
undertaken by others
Review manuscripts in your area of research
Inform your own research
Conduct your own research
The virtuous circle of peer review and
research community engagement
![Page 23: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/23.jpg)
Reviewer accountability 1:A creditable activity
23Spectrum (Official newsletter of the Australian Institute of Radiography), November 2014
![Page 24: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/24.jpg)
Reviewer accountability 2:Public recognition
24
showcaseshowcase
get creditget credit
![Page 25: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/25.jpg)
Reviewer accountability 3:The Hippocratic Oath of peer review
25
transparencytransparency reproducibilityreproducibility
![Page 26: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/26.jpg)
Common themes for authors
26
transparencytransparency
rigourrigour
speedspeed
collaborationcollaboration
![Page 27: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/27.jpg)
‘Aim high, publish fast’
27
![Page 28: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/28.jpg)
Peer review variables
Transparency
Timing
Submission process
Assessment of…
Reviewerselection
Decision process
-Open peer review-Closed peer review
-Pre-publication-Post-publication
-Direct to a journal-To a 3rd party-Transfer within “family”
-Science only-Science + novelty or impact
-Independent review-Interactive review -Collaborative review
-Authors recruit reviewers-Editors recruit reviewers-Reviewers “bid” for papers
![Page 29: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/29.jpg)
‘…publishers would welcome more guidance from key sections of the research community on the kinds of peer review services they want publishers to provide, and on the purposes that they should seek to fulfil.’
Scholarly Communication and Peer Review: The Current Landscape and Future Trends, Research Information Network
CIC, March 2015
Peer review evolution:A collaborative effort
29
![Page 30: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/30.jpg)
The journey of peer review evolution
30
Listen to the community
Work with the community
Do more research
Be flexible
Train, educate, evaluate
Collaborate with stakeholders
![Page 31: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/31.jpg)
Collaborate with stakeholders
31
![Page 32: UKSG Conference 2015 - “Peer review is dead, long live peer review.” Michael Willis Wiley](https://reader030.vdocuments.mx/reader030/viewer/2022032420/55a5f21e1a28abce3d8b4637/html5/thumbnails/32.jpg)
32