The Peer Review Process.

Why Evidence Based Practices Are Needed?

Isabelle Boutron

METHODS team

Research Centre of Epidemiology Biostatistics Sorbonne Paris Cité

Paris Descartes University

Methods in Research on Research

Acknowledgements

MiRoR projectJoint doctoral training program, to train 15 PhD students in Methods in Research on Research in the field of clinical research (http://miror-ejd.eu) funded by Marie Skłodowska-Curie Actions.

Projects-Planning-Conduct -Reporting -Peer review

7 European Universities and 10

International Partners

The peer review system

• Central to the scientific community

– Gatekeeper of the scientific publications

– To improve the quality of manuscripts

• The number of journals and manuscripts is increasing

– 8 million researchers publishing 2.5 million articles in 28,000 peer-

reviewed English-language journals/year

– The annual revenue is $10 billion

• A system relying mainly on work performed voluntarily by

academic researchers

3International Association of Scientific, Technical and Medical Publishers STM Report March 2015

The peer review system

2.7 billion US

dollars / year*

*Look H, Summary report: Rightscom Ltd for JISC Collections; 2010

The performance of the system is

questioned.

5

« material often obtained in the most rigourous kind

of biomedical experimentation is rated as publishable

or not by a system that has rarely been subjected to

any analysis, let alone one that is rigourous »

Franz Ingelfinger, editor of the NEJM (1967 to 1976)

Level of evidence of peer review

6

Published in Issue 2, 2008

At present, little empirical evidence is available to

support the use of editorial peer review as a

mechanism to ensure quality of biomedical

research.

The performance of the system is

questioned.

7

Submission of 304 versions of a fake paper with fatal flaw to open-access journals.

• More than half of the journals accepted the paper, failing to notice its fatal

flaws.

• For DOAJ publishers, that completed the review process, 45% accepted the

paper.

Impact of peer review on reporting of

RCTs

• Comparison of the submitted and accepted version of manuscripts

of RCTs published in BMC journals

• Changes were limited

– A median of 11% (range 1-60%) words deleted and 20% added

(range 2-88%).

• Peer reviewers often fail to detect important deficiencies in the

reporting

• Peer reviewers requested relatively few changes for reporting

8Hopewell S, BMJ, 2015

Detection of selective reporting of

outcomes

9

34% examined information registered on a trial

registry.

Can peer reviewers reduce spin ?

• Assessment of the manuscript submitted, peer reviewers comments, and final manuscript of non-randomized studies assessing a therapeutic intervention published in BMC Series journals (2011-2013).

• 55% of submitted manuscripts, peer reviewers identified at least one example of spin

• Of the spin identified by peer reviewers – 67% were completely deleted,

– 16% partially deleted

– 17% not removed in the final published article.

• For 15%, peer reviewers requested adding some spin

10Lazarus, J Clin Epidemiol. 2016

Level of spin before and after peer review in

the abstract conclusion

11

Before peer-review After peer-review

High

Moderate

Low

No

N=66 N=61

N=24 N=24

N=10 N=17

N=28 N=26

76% Peer reviewers failed to identify spin in abstract conclusions

Lazarus, J Clin Epidemiol. 2016

Two versions of a well-designed randomized controlled trial that differed only in the direction of the finding of the principal study end point

238 reviewers at 2 journals (assigned at random)

• were more likely to recommend the positive version of the test manuscript for publication than the no-difference version (97% vs 80%, P<0.001)

• detected more errors in the no-difference version than in the positive version (mean 0.85 vs 0.41, P<0.001)

• awarded higher methods scores to the positive manuscript than to the (identical) no-difference manuscript (8.24 vs 7.53, P=0.005)

12

Influence of authors’ prestige

• 119 reviewers were randomized to assess a fabricated

manuscript with the prestigious authors’ names and

institutions masked or visible

– Reviewers were more likely to recommend acceptance when

the prestigious authors’ names and institutions were visible than

when they were redacted

• 87% vs 68%;

• RR, 1.28 [95% CI, 1.06-1.39], P = .02

– They gave higher ratings for the methods.

13

The process is not clearly defined and

standardized

• Who are the reviewers?

– Who can be a reviewer?

– How are they identified?

– What are the core competencies?

– Should reviewers be trained and how?

• What documents should be peer-reviewed?

– Manuscript

– Appendices

– CONSORT checklist

– Registry

– Protocol, SAP

– …

– CSR?

– Raw data?14

• The tasks expected from a peer-reviewer are not realistic

– More than 200 different tasks identified

– The tasks involved different level of expertise and different

backgrounds

• Statistical and methodologic expertise

• Content expertise (novelty, relevance, external validity etc)

• None

– Verification (adherence to guidelines, consistency with registries)

– Formatting

15

Multiple tasks asked to reviewers

(important tasks for reviewers are not congruent

with important tasks for editors) Chauvin, BMC Medicine, 2016

• The most important task

for peer reviewers

(evaluating the risk of

bias) was clearly

requested by 5 % of

editors.

• The task most frequently

requested by editors

(provide recommendations

for publication), was rated

in the first tertile only by

21 % of all participants.

17

Several interventions are implemented

or proposed to improve the system

18

19

What is the evidence?A Systematic Review and Meta-analysis of Interventions to Improve the

Quality of Peer Review

• Only 22 reports of RCTs

• Only 7 were published over the past

10 years

• Interventions assessed

– Blinding

– Open peer review

– Training

– Use of checklist

– Adding experts

• Most are performed in one specific

journal

• Low methodologic quality

Bruce, BMC Med, 2016

RCTs and Peer review

• There are no barrier to the conduct of RCTs

• Large sample size is available (manuscripts, reviewers)

• Consent with an opt out approach facilitate recruitment

• Randomisation, allocation concealment are easy to perform

• Some outcomes are routinely collected

• Blinding outcome assessment is easily feasible

21

RCTs and Peer review : Methodological

research is needed

• What is the best study design for assessing interventions to improve the

peer review process?

– Design: RCTs, cluster RCTs, pairwise comparisons, Stepped wedge cluster RCT

of journals, time series analysis etc

– Unit of randomisation

• Manuscripts

• Reviewers

– Type of manuscript (real, fabricated)

• What will be the outcome?

– Quality of the peer review report (editor’s subjective assessment, validated scales

to assess the quality of the peer review report

– Quality of the final manuscript (how is high quality defined, who is to decide?)

– Need of CORE outcome set

• Ethical issues

– Impact on the final decision

Modeling (Agent-based model)

To approach the complexity of the scientific publication system

and compare different systems

Kovanis Plos One 2016

Kovanis Scientometrics. 2016

Explore

sustainability

of the system

Compare

different

systems

(cascade etc)

24

• A call to substantially increase the amount of research conducted by journals

in journalology and meta-research.

– Participate in Randomized Controlled Trials (eg about peer review

improvement)

– Share their empirical data to help modeling

– Share manuscripts submitted, reviews, revised manuscripts to develop

qualitative and quantitative research on the impact of peer review

Conclusions

25

• “So peer review is a flawed process, full of easily

identified defects with little evidence that it works.

Nevertheless, it is likely to remain central to science

and journals because there is no obvious alternative.”

• “The most important question with peer review is not

whether to abandon it, but how to improve it.”

(R. Smith)

“We need rigorous studies to tell us the pros and cons of

these approaches.” (D Rennie)