3.4-2018 cse prd slides final...5/14/2018 6 double-blind peer review: strengths • reduce/eliminate...
TRANSCRIPT
5/14/2018
1
Melissa Blickem
• Senior Peer
Review Analyst,
American Chemical
Society
Margaret Donnelly
• Publisher, IOP Publishing
Lindsay Morton
• Contributor Experience Manager, Public Library of Science
SabinaAlam
• Editorial Director, F1000 Platforms
Moderator: Jonathan Schultz
• American Heart Association
Concurrent #3.4: Peer Review Debate#CSE2018
Peer Review Debate:Single-Blind vs Double-Blind vs
Open vs Post-Publication
Moderator:
Jonathan Schultz
American Heart Association
Blinded Peer Review
• Single-Blind:
– Reviewers know identity of authors
– Reviewer identity unknown to authors & public
• Double-Blind:
– Reviewers do not know identity of authors
– Reviewer identity unknown to authors & public
5/14/2018
2
Unblinded Peer Review
• Open/Transparent:
– Identity of authors & reviewers known throughout the review process
– Reviewer names & reviews typically published with articles
• Post-Publication:
– Articles posted with minimal or no review
– Review occurs in the open with names and reviews visible to authors & public
The Debate
• Goal: Determine which format is the “best” at
generating high-quality research articles in a
practical and sustainable way.
• Format: Brief overview of each format,
followed by Q&A/panel discussion
– Questions from the audience are encouraged
Our Debaters
• Single-Blind: Melissa Blickem, Senior Peer Review
Analyst, American Chemical Society
• Double-Blind: Margaret Donnelly, Publisher, IOP
Publishing
• Open/Transparent: Lindsay Morton, Contributor
Experience Manager, Public Library of Science
• Post-Publication: Sabina Alam, Editorial Director,
F1000 Platforms
5/14/2018
3
American Chemical Society
Single Blind Peer Review
Melissa BlickemLead Peer Review Analyst, Peer Review Services
Editorial Services, Global Journal DevelopmentACS Publications
Peer Review Debate: Double-blind vs Single vs Open vs Post
2018 Council of Science EditorsMay 7, 2018
Peer-Review Survey of Journal Authors and Reviewers
Main purpose - to investigate comments about perceived bias emerging from India trip
Sent to 15,741 total authors/reviewers
Total finished survey: 2,246 (14% response rate)
75% non-US based authors/reviewers, 66 countries represented
For research papers published in your field, to what extent do you agree that the following
types of peer review are/would be effective?
4.2% 4.7%
33.5% 33.5%39.6%
19.6%6.8% 8.8%
26.5% 26.5%24.1%
19.3%
11.7%
20.1%
21.7% 23.5%22.9%
33.0%
31.1%
30.0%
12.5% 11.7%9.8%
20.3%46.2%
36.4%
5.8% 4.9% 3.6%7.8%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Single-blind peer
review
Double-blind peer
review
Open peer review Open & published
peer review
Post-publication open
commentary
Blind peer-review
supplemented with
post-publication open
commentary
Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree
5/14/2018
4
2.6% 4.7%
31.6%35.7%
38.5%
22.1%
3.0%5.1%
19.4%
21.6%19.2%
16.0%
6.5%
11.4%
22.7%
23.7%26.7%
29.6%
24.3%
29.5%
16.5%
12.7%10.7%
20.2%
63.6%
49.2%
9.9%6.4% 4.9%
12.1%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Single-blind peer
review
Double-blind peer
review
Open peer review Open & published
peer review
Post-publication open
commentary
Blind peer-review
supplemented with
post-publication open
commentary
Not Willing Somewhat Unwilling Neutral/Don't Know Somewhat Willing Very Willing
As a reviewer, to what extent are you/would you be willing to
participate in the following types of peer-review?
2.7% 3.9%
25.1%31.5%
37.3%
21.0%
4.4% 4.7%
15.4%
19.8%
20.3%
16.6%
8.2%12.1%
25.3%
25.1%
25.9%
30.4%
28.0%
30.2%
20.2%
15.7%
11.9%
21.7%
56.7%
49.0%
14.0%7.9%
4.7%10.3%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Single-blind peer
review
Double-blind peer
review
Open peer review Open & published
peer review
Post-publication open
commentary
Blind peer-review
supplemented with
post-publication open
commentary
Not Willing Somewhat Unwilling Neutral/Don't Know Somewhat Willing Very Willing
As an author, to what extent are you/would you be willing to
participate in the following types of peer- review?
In Conclusion
Single-blind peer-review perceived as most effective by reviewers (77% agree)
• Double-blind peer-review also perceived as effective by reviewers (66% agree)
• Open peer-review and post-publication peer-review perceived as not effective.
5/14/2018
5
American Chemical Society
Single Blind Peer Review
Melissa BlickemLead Peer Review Analyst, Peer Review Services
Editorial Services, Global Journal DevelopmentACS Publications
Peer Review Debate: Double-blind vs Single vs Open vs
2018 Council of Science EditorsMay 7, 2018
Double-blind peer review
2018 CSE Annual Meeting
May 7, 2018
Margaret Donnelly
Publisher
Introduction
• Double-blind peer review– The identities of the authors are
blinded to the reviewers
– The identities of the reviewers are not
made public with the publication of
the paper
– IOP Publishing led a year-long pilot on
double-blind peer review on its
Express titles – Materials Research
Express (MRX) and Biomedical Physics
& Engineering Express – “inclusive”
international journals
5/14/2018
6
Double-blind peer review: strengths
• Reduce/eliminate bias in the peer review process– Studies have shown that blinded peer review reduces bias against certain
authors, e.g., female authors, junior authors, controversial authors, authors
from less prestigious institutions, etc. (1)
– Similarly, other studies have shown that referees who know the identities of the
authors are more likely to recommend acceptance for papers from famous
authors and authors from top universities and companies than referees who do
not know the identities or institutions of the authors (2)
Double-blind peer review: strengths
• Potential to improve referee acceptance rates– When the identities of authors are known to potential referees, a 2017 PNAS
study found that referees accept invitations to review 22% fewer papers than
referees who are blinded to the authors’ identities (2)
– In double-blind peer review, referees may be more inclined to agree to review
a paper based on personal interest rather than factors like the authors’
identities or institutions
– There is evidence that referees decline to review at higher rates for journals
with open peer review compared to journals using blinded peer review (3),
leading to longer turnaround times
Double-blinded peer review: strengths
• Referees have the freedom to provide
honest feedback– Referee anonymity ensures that reviewers can
provide critical (relevant, important) feedback
without fear of retaliation
– The critical feedback provided by double-blind
peer review leads to higher rejection rates, and
subsequently higher quality published articles
– Several studies have shown that double-blinded
referees recommend rejection at a higher rate
than referees whose identities are known to the
authors (4,5)Kowalczuk et al. BMJ Open 2015
5/14/2018
7
Summary
• Double-blind peer review is the best format of peer review:
– It reduces bias and is fairer than single-blind
– It encourages more objective, critical reviews, leading to higher
rejection rates and higher quality published articles
– It is the most popular system with referees, leading to faster
publication times
References1. Triggle CR, Triggle DJ. What is the future of peer review? Why is there fraud in science? Is plagiarism out of
control? Why do scientists do bad things? Is it a case of: “all that is necessary for the triumph over evil is that
good men do nothing”? Vasc Health Risk Manag 2007; 3:39-53.
2. Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single- versus double-blind peer review. PNAS 2017;114:
12708-12713
3. Kowalczuk M, Samarasinghe M. Comparison of Acceptance of Peer Review Invitations by Peer Review Model:
Open, Single-blind, and Double-blind Peer Review. Presented at the International Congress on Peer Review and
Scientific Publication, September 10-12, 2017. https://peerreviewcongress.org/prc17-0227
4. Kowalczuk MK, Dudbridge F, Nanda S, et al. Retrospective analysis of the quality of reports by author-suggested
and non-author-suggested reviewers in journals operating on open or single-blind peer review models. BMJ
Open 2015;5:e008707
5. Vinther S, Nielsen OH, Rosenberg J, Keiding N, & Schroeder TV. Same review quality in open versus blinded peer
review in “ugeskrift for Laeger. Dan Med J 2012; 59:A4479
6. Snodgrass RT. Single- versus double-blind reviewing. ACM Trans Database Syst (TODS) 2007: 32:1-29
Transparent Peer Review
Lindsay Morton
Contributor Experience Manager, PLOS
May 2018
5/14/2018
8
What is Open Peer Review?
No universally accepted definition. For purposes of
discussion, we are referring to peer review in which:
⊨ Author, editor, and reviewer identities are known to all
participants throughout the peer review process
◇ Reviewer and editors names appear alongside the
published article
◇ Reviewer comments posted alongside the published article
Opt-in Approaches
Optional open peer review
scenarios
Reviewer name
disclosed to
authors
Reviewer name
published
Reviewer report
published
Author and reviewer both agree to open peer review
✘ ✘ ✘
Author agrees to open peer review but reviewer does not
✘
Reviewer agrees to open peer review but author does
not
✘
No one agrees to open peer review
Why Choose Open Peer Review?
1. Accountability
2. Quality
3. Reviewer credit
4. Enrich scientific record
5. Momentum and innovation
5/14/2018
9
1. Accountability
Each participant in the peer review process is accountable
for their own assessment and remarks
� Ethical benefits:
� Competing interests are more apparent
� Discourages malicious comments, plagiarism, and pursuit
of personal agenda
� Qualitative benefits:
� Encourages thoroughness, courtesy, and fairness
2. Quality
Evidence is not conclusive, but studies suggest that open
peer review feedback is as good as, or slightly better than,
closed models
� More constructive
� More in-depth
� Longer
� Attention to methods
� Substantiating evidence underlying reviewer comments
3. Reviewer credit
Reviewers are hardworking volunteers who want and deserve
acknowledgement. Posting peer reviews publicly enables:
• Enhanced presentation of reviewer activity for professional
records
• Opportunities for reviewers to demonstrate their expertise
• Recognizing contributions of students or colleagues
assisting with review
• Future elevation of review reports to citable scientific
outputs in their own right
5/14/2018
10
When reviews are open to the wider scientific community,
readers:
• Have access to more expert scientific opinions
• Have context for interpreting results
• Have model reviews they can emulate
• Can easily see whether their own questions were
previously addressed in review
4. Enrich scientific record
Open peer review is gaining ground among stakeholders
across the scientific publishing landscape:
• Many journals already mandate or offer an Open Peer
Review option
• Authors like signed peer review
• Reviewers are open to signed and posted reviews
5. Momentum and innovation
Open Invited Peer Review,
After Publication
Sabina Alam
Editorial Director, F1000 Platforms
5/14/2018
11
• Speed of publication
• Transparent (peer review and
data sharing)
• Credit for reviewers (DOI for
reports and ORCID IDs)
THE F1000RESEARCH PUBLISHING AND PEER REVIEW PROCESS
Scope
Language
Reporting guidelines
Data availability
Ethics
Reviewer suitability
(competing interests,
expertise, etc)
Methods & analysisStrength of conclusionsScientific validity
~ 7 days
DOI
DOI
TRANSPARENT REFEREEING AND REVIEW STATUS
https://f1000research.com/articles/6-1698/v3
Indexed once it passes peer review:
or
5/14/2018
12
http://f1000research.com/articles/4-
121
TRANSPARENT REFEREEING AND DISCUSSION
Referees:
� Get credit for contributing to discussion
�Focus on helping authors improve their work
�Their reports provide new form of expert
article-based assessment
Open invited peer review, after publication
The advantage of the open refereeing process becomes completely obvious once you
have experienced it. When referees comment openly on a paper that is already
published, they suddenly become much more constructive and helpful.
Etienne Joly
IPBS, CNRS, Université de Toulouse, France
The open review process pushes authors to submit an
accurately prepared manuscript and reviewers to
provide constructive comments.
Gianmarco Contino
MRC Cancer Unit, University of Cambridge, UK
In my opinion, this publication model represents the future of scientific
publishing. Review is completely transparent to all involved, tyrannical
reviewers are avoided, authors drive the process, it is open access and the
speed is amazing.
Eleftherios Diamandis
Mount Sinai Hospital, Canada
Panel Discussion
• Single-Blind: Melissa Blickem, Senior Peer Review
Analyst, American Chemical Society
• Double-Blind: Margaret Donnelly, Publisher, IOP
Publishing
• Open/Transparent: Lindsay Morton, Contributor
Experience Manager, Public Library of Science
• Post-Publication: Sabina Alam, Editorial Director,
F1000 Platforms