editorial board review a few good reviewers don schneider, ph.d
DESCRIPTION
Editorial Board Review A Few Good Reviewers Don Schneider, Ph.D. CSR Advisory Council Meeting May 19, 2014. Toward Review by the Best. Reviewer pools Past experience Cost considerations. Reviewer Pools. Reviewer StatusNumbers HHMI300 NIH R37700 NAS2,000 R0126,000 - PowerPoint PPT PresentationTRANSCRIPT
CSR Advisory Council Meeting
May 19, 2014
Editorial Board ReviewA Few Good Reviewers
Don Schneider, Ph.D.
• Format modeled on journal manuscript review
• First piloted in 2008 with 6 SBIR panels, just in time for TR01s, Challenge grants, DP1 etc.
• Two stages– First Stage – Mail reviewers– Second Stage – Editors
Past Experience
First Stage/Mail Reviewers
• Subject matter experts– Provide depth in review
• Focus on scientific and technical merit• 2-3 first stage mail reviewers per application• Submit full critiques• Give overall impact and criterion scores
– Overall impact scores not factored into final priority score
• Hold face-to-face meeting• Recruit broad experts
– Provide perspective in review (assign about 15 applications each)
• Focus on impact and significance• Assign 3 second stage reviewers per application• Consider first stage critiques in review• Write overall impact paragraph• Give overall impact score
– Final priority score based on second stage only
Second Stage/Editors
• Provides both depth and breadth in review
• Optimizes use of the best reviewers
• Scales well for large numbers of applications(second stage discusses a fraction of the applications)
Rationale
Perceived Advantages
• Involves no travel/teleconference for first stage reviewers
• Allows small, interactive face-to-face meetings
• Promotes better scoring and assessment of impact
• Lessens travel and lodging expenses and inconveniences
Review # of Applications Cost/application
Regular R01 F2F $518
DP1/Pioneer EB+I 244 $280
DP2/New Innov EB 593 $124
DP5/Early Indep EB+I 84 $875
Cost Considerations(Alicia Caffi)
• Recruitment of large numbers of reviewers
• Timeline– Tight, two sequential reviews – (in the 17 week cycle)
• More staff time required (SROs)
• Some sense of isolation by first stage reviewers
Challenges
• Each application examined by at least 5 reviewers
• Interactive, thoughtful discussions
• Overall scoring by second stage members
• Reviewers and staff like final review products
Review Outcomes
• Survey conducted by A Kopstein of reviewers participating in SBIR pilots 2008
• Outcomes were generally positive
– Majority willing to participate in either review stage in future
– Editorial Board Review:• Increases expert review
» 3/4ths of respondents• Preferred for their own applications
» 2/3rds of respondents
Survey