uksg conference 2015 - “peer review is dead, long live peer review.” michael willis wiley

‘Peer review is dead: long live peer review!’

Michael WillisWiley, Oxford, UK

miwillis@wiley.com @ctyerkes

1

A venerable legacy

2

"Philosophical Transactions Volume 1 frontispiece" by Henry Oldenburg - Philosophical Transactions. Licensed under CC BY 4.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:Philosophical_Transactions_Volume_1_frontispiece.jpg

Augustus Bozzi Granville" by Unknown - Wellcome Images. Licensed under Public Domain via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:Augustus_Bozzi_Granville.jpg#mediaviewer/File:Augustus_Bozzi_Granville.jpg

A venerable tradition

3

• Augustus Bozzi Granville (1783-1872)

• First medical autopsy on an Ancient Egyptian Mummy, described to the Royal Society in 1825

• Critique of peer review as conducted by Royal Society

Science without a head; or, the Royal Society dissected

London, 1830

The Royal Society in the 19th century; being a statistical summary of its labours during the last thirty-five years, with many original tables and official documents (never before published)

London, 1836

Polemic against the old order

4

Peer review is…

Arbitrary

‘Much oftener is the fate of a paper committed to the chances of the mere yea-and-nay box, than to the decision of a competent judge instructed to offer a preliminary opinion upon its merits.’

Incompetent

‘At many of these meetings, members of the committee of papers have been present who have not the smallest pretension to any knowledge whatever of the subject under consideration.’

Critique of method

5

‘It has been said that the warp that holds the complex fabric of science together is peer review, and the woof is the noise made by scientists who complain about it.’

Jukes, TH. Peer review. Nature 265 (1977); 203

‘Those who complain of unfair treatment at the hands of reviewers would do well to remember that 400 years ago they would have had—at best—the choice between the burning of their manuscripts or of themselves.’

Grivell, L. Through a glass darkly: The present and the future of editorial peer review. EMBO reports 7 (2006); 567-570

Leaping forward…

6

‘…slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless for detecting fraud…’

Smith, R. Peer review: reform or revolution? BMJ 315 (1997): 759–760

A damning indictment

7

Nails in the coffin of peer review?

8

It’s a black box

It’s a black box

There are too few reviewers to do the work

There are too few reviewers to do the work

It takes too longIt takes too long

It costs too muchIt costs

too much

Microsoft Clip ArtIt’s not rigorous enough

It’s not rigorous enough

The web changes

everything

The web changes

everything

How strong are the nails?

9

Why does this matter?

furthering of scientific

endeavour

health and welfare of society

public and academic

confidence in scientific record

integrity of scientific record

10

‘Many forms of Government have been tried, and will be tried in this world of sin and woe. No-one pretends that democracy is perfect or all-wise. Indeed, it has been said that democracy is the worst form of Government except all those other forms that have been tried from time to time.’

Peer review resurgens

11

peer reviewpeer review

traditional peer review

traditional peer review

traditional peer review

traditional peer review

peer reviewpeer review

"Sir Winston S Churchill" by United Nations Information Office, New York - Library of Congress, Reproduction number LC-

USW33-019093-C© US Federal Government

Peer review: alive and kicking

12

clipartp

anda.com

‘…peer review is not only alive and kicking, but apparently increasing its influence, despite the many potential (or invented) threats posed by a rapidly unfolding and enveloping digital environment…’

Peer review: still king in the digital age, Nicholas et al., Learned Publishing 28(2015):15-21

Peer review under increased scrutiny

13

—Research indicates that the vast majority of researchers still value and trust peer review

—More scholars are looking at peer review as a scientific object of study

—There are now more organisations devoted to understanding and improving peer review than ever before

—There are several new products and services designed to preserve and enhance peer review

‘Four reasons to feel good about the future of peer review’

14

Eric Hallhttp://www.researchinformation.info/news/news_story.php?news_id=186210 March 2015

Peer review in scientific publications (July 2011)http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/856.pdf

‘…despite the many criticisms and the little solid evidence on its efficacy, editorial peer review is considered by many as important and not something that can be dispensed with.’

‘In order for current peer-review practices to be optimised and innovative approaches introduced, publishers, research funders and the users of research outputs (such as industry and government) must work together.’

‘…we encourage the prudent use of online tools for post-publication review and commentary as a means of supplementing pre-publication review.’

‘The integrity of the peer-review process can only ever be as robust as the integrity of the people involved.’

UK Parliament on peer review

15

Peer review survey, Sense About Science (2009)http://www.senseaboutscience.org/data/files/Peer_Review/Peer_Review_Survey_Final_3.pdf

—What do you think of peer review overall?

—What value does peer review add to research output?

—How can peer review be improved?

Researchers on peer review

16

Researchers on peer reviewWhat do you think of peer review overall?

17

Researchers on peer reviewWhat value does peer review add?

18

Researchers on peer reviewHow can peer review be improved?

19

Joynson C and Leyser O 2015 [v1; ref status: approved 1, http://f1000r.es/53j] F1000Research 2015, 4:66 (doi: 10.12688/f1000research.6163.1)

The culture of scientific research, Joynson C and Leyser O.Conclusions of 2014 survey by Nuffield Council on Bioethics

Researchers on peer reviewTraining: an opportunity for publishers

21

Find out what others are doing in your area of

research

Critique the research

undertaken by others

Review manuscripts in your area of research

Inform your own research

Conduct your own research

The virtuous circle of peer review and

research community engagement

Reviewer accountability 1:A creditable activity

23Spectrum (Official newsletter of the Australian Institute of Radiography), November 2014

Reviewer accountability 2:Public recognition

24

showcaseshowcase

get creditget credit

Reviewer accountability 3:The Hippocratic Oath of peer review

25

transparencytransparency reproducibilityreproducibility

Common themes for authors

26

transparencytransparency

rigourrigour

speedspeed

collaborationcollaboration

‘Aim high, publish fast’

27

Peer review variables

Transparency

Timing

Submission process

Assessment of…

Reviewerselection

Decision process

-Open peer review-Closed peer review

-Pre-publication-Post-publication

-Direct to a journal-To a 3rd party-Transfer within “family”

-Science only-Science + novelty or impact

-Independent review-Interactive review -Collaborative review

-Authors recruit reviewers-Editors recruit reviewers-Reviewers “bid” for papers

‘…publishers would welcome more guidance from key sections of the research community on the kinds of peer review services they want publishers to provide, and on the purposes that they should seek to fulfil.’

Scholarly Communication and Peer Review: The Current Landscape and Future Trends, Research Information Network

CIC, March 2015

Peer review evolution:A collaborative effort

29

The journey of peer review evolution

30

Listen to the community

Work with the community

Do more research

Be flexible

Train, educate, evaluate

Collaborate with stakeholders

Collaborate with stakeholders

31

32