Welcome to the 10th Annual Meeting of ISMPP!

Leading Through Collaboration

1

Leveraging Peer-Review Comments

Lorna Fay and Laverne A. MooneyPublications Management Team

Pfizer, Inc.

Background

Rationale for study

• Acceptance rates at first choice journals ~50-60%

• Rewriting, resubmission, subsequent peer review delaying data to public by 3-4 months

• If evaluating and leveraging the criticisms in these reviews could improve the manuscript development process, it could save time and resources

3

Objective• Our goal was to assess whether it is possible to

leverage journals’ peer-reviewer comments:– speed the delivery of clinical trial results to the

public– to improve acceptance rates– increase efficiency

4

Study Sample: Rejection Letters from Primary Manuscripts* - Studies Completed in 2010

5

Manuscripts for Interventional Studies in Patients that Completed in 2010

Rejected from 1st choice Journal (n=37) 

Accepted at 1st choice journal 

Rejection Letter NA  (N=3)

Rejected With Peer‐Review 

(N=18 )

Rejected With No Peer‐Review 

(N=16 )

*Primary manuscripts = manuscripts describing the primary endpoint of the study. Total of 37 manuscripts resulted from 36 studies; 1 study had 2 primary manuscripts

Development of Rejection Letters (RL) Issues Scoring System

A scoring system was developed by identifying common issues in the rejection letters and then categorizing them

Nine types of issues commonly observed. These were the final categories against which the RL were to be tallied

Category of Issues*

A.  Study not novel

B.  Questionable rationale

C.  Study design limitations

D.  Methods description

E.  Statistical analysis

F.  Industry involvement

G.  Adverse event details

H.  Conclusions not supported by data

I.  Sample size

*Not in order of importance or frequency

Research Design and Methods

• Read each rejection letter– Recorded date of rejection letter

– Assessed whether manuscript had undergone peer-review or not

• For manuscripts that had undergone peer review:– Recorded number or peer-reviewers

– Scored the peer-reviewers comments using the pre-determined categories

– Each rejection letter was read and scored by 2 reviewers independently and any disagreements reviewed

• Obtained submission date to journal from Datavision

7

8

Rejection Letter with Reviewers’ Comments

Typically 2 peer-reviewers. Range 1-3

Implementation of Rejection Letter Scoring System

• Each issue was marked with a letter A, B, C… etc. – For example, “A “was marked in the RL next to comment about

insufficient details about methods.

– “B” was marked when study design limitations were discussed in the RL

– Each issue was tallied only once per manuscript even though it may have been mentioned multiple times

10

Put in scoring

Results

11

Results: Most Common Reasons for Rejection

2

4

5

6

7

8

9

11

14

0 2 4 6 8 10 12 14 16

Industry involvement

Questionable rationale

Study not novel

Adverse event details

Sample size

Conclusions not…

Statistical analysis

Study design limitations

Methods

Number of Rejection Letters Mentioning Issue N = 18 

Conclusions not supported

Journal Rejection Letters for 36 studies: Peer Review (Yes/No) by Phase of Study

13

3

5 5

3

1

2

8

7

0

1

2

3

4

5

6

7

8

9

Phase 1 Phase 2 Phase 3 Phase 4No Peer Review N=16 Peer Review N=18

Late phase study manuscripts more likely to get external peer-review

14

Time to Rejection by Peer-Review

7.5

56.5

0

10

20

30

40

50

60

No Peer Review N=16 Peer Review N=18

Median Time to Rejection (days)

Range: 19 to 330*Range: 2 to 81

*Time including ongoing discussions with journal

15

Impact Factor of Journals by Peer-Review

6.9195

5.1885

0

1

2

3

4

5

6

7

8

No Peer Review N=16 Peer Review N=18

Median IF

Range: 4.16 to 51.658 Range: 2.601 to 51.658

Summary

• Half of rejected manuscripts in this sample were rejected without peer-review from the first choice journal

• Most frequently identified rejection reasons– Methods needed clarification

– Study design limitations

– Statistical analysis

• Median time to decision was longer when manuscript sent for peer-review vs manuscripts rejected outright (54 vs 8 days)

16

Conclusions

• Appear to be opportunities to leverage the peer-review comments to implement training to improve overall quality

• Issues that can be dealt with during manuscript writing:– Ensuring methods are sufficiently detailed

– Check that conclusions are supported by data, not overstated

– Reporting adverse events

Limitations– Small descriptive study

– Often multiple reasons for rejection

17

Next Steps

• Review more rejection letters to increase sample size

• Develop training and tips

• Share tips with authors, teams and agencies

• Continue to educate authors re appropriate journal selection

Thank You

19

Welcome to the 10th Annual Meeting of ISMPP!

Leading Through Collaboration

20