Scientific Reviewing for Physicians: a CLEAR Approach

John Iskander MD MPHCAPT, U.S. Public Health Service

Senior Medical ConsultantOffice of the Associate Director for ScienceCenters for Disease Control and Prevention

“What do you read, my Lord?” “Words, words, words…”-Hamlet, William Shakespeare

Outline

•Importance of scientific reviewing

•Types of scientific reviews

•The CLEAR framework

•Categorizing comments

•Skill-building, resources, and training

•Questions

Importance of Scientific Review

•Essential to the scientific enterprise•Building scientific credentials

•Author, reviewer, editorial board member•Collaborating with colleagues•Supporting science and the PHS mission

•Little or no formal instruction as part of medical training•An underappreciated aspect of scientific leadership•Not much in the “peer reviewed” literature on how to do a peer review!

USPHS Mission• The mission of the U.S. Public Health Service

Commissioned Corps is to protect, promote, and advance the health and safety of our Nation. As America's uniformed service of public health professionals, the Commissioned Corps achieves its mission through:

• Rapid and effective response to public health needs

• Leadership and excellence in public health practices

• Advancement of public health science

Basis for this Talk

•“Practice-based evidence” building on experience of serving as a peer reviewer, clearance reviewer, and policy reviewer

•OADS fellowship, scientific reviewing curriculum•Short-term rotations support the professional development of a scientist interested in pursing a career track in the science leadership, management and policy field

•Discussions with and feedback from colleagues

Types of Scientific Review•Subject matter expert (SME)

•Example: a peer review requested by a journal editor (or internally by a colleague)•Is content scientifically valid, of importance, novel?

•Policy review•Example: clearance or agency approval•Is content consistent with policies and programs?

•Editorial review•Example: Pre-publication review of an article or book chapter by editor•Can clarity be improved without major changes to content?

Typical 3-Level Comment Framework

•Level 1 or mandatory•Example: serious errors of fact or data interpretation, misstatement of agency recommendation or policy

•Level 2 or “revise or respond”•These should be revised to improve understanding, but the authors can explain why revision is not possible (e.g. data to answer question is not available)

•Level 3 or editorial•Often presented as inline edits•If any such changes are mandatory, make sure this is clearly labeled

Before You Start

•Clearly understand:•Your role, the type of document you are reviewing, and the type of review requested•“Who are you, and what is this?”

•Respect the effort that went into creating the document

•Meet deadlines

The CLEAR Framework

•Clarity

•Logic

•Ethics

•“Agency”

•Relevance

What is Clarity?

“Omit needless words.”-Strunk and White, The Elements of Style

• Shorter sentences, shorter (or no) parenthetical

statements

• Consistent “voice”

• Do ideas presented conflict with each other or appear contradictory?

What is Logic?

• Flow of ideas

• Being able to follow an article’s “argument”

• Interpretation of findings that is in agreement with data presented

• Reasonable inferences drawn by authors about explanations for findings

• Should hold both within paragraphs and sections, between them, and for the document as a whole

What is Ethics?

•Not only whether there is documentation of IRB clearance and/or clinical trial registration, but also whether readers will perceive an ethical issue with the work that was done

•Vulnerable populations? •Informed consent?•Privacy/confidentiality concerns?

What is “Agency”?

•Dual meaning:

•Who is making the recommendations? Individuals, agency?

•In other words, who is the “agent”?

•Can the agency stand by the science and policy contained in the work?

•Of special importance for policy reviews

What is Relevance?

•“Information for action”•What are the authors asking for-awareness of emerging or rare conditions, use of new diagnostic criteria or laboratory tests, reporting to public health authorities, some or all of these?

•Practical value to clinicians•Rather than saying, “clinicians should look out for this incredibly rare disease” say “for patients with risk factors, this disease should be considered and the following laboratory tests may be obtained”

What Makes a Good Review?

•Feedback that is constructive, specific, and can be acted on by the author to improve some aspect of CLEAR

•Not: “This paragraph needs to be tightened up a little bit”

•Instead:“The second and third sentences can be eliminated as they duplicate information presented elsewhere”

What Makes a Good Review?

•A brief (2-3 sentence) summary of the review may be appropriate

•Might be only for your use, or may be shared with authors and/or editors•Crystallizes your thinking, and points out “big picture” issues

•Do not rewrite extensively unless you are co-authoring or editing

•Recommending alternative language for a sentence or part of a sentence often appropriate (choose your words carefully-you may see them in print)

What Makes a Good Review?

•Be very clear about what must be changed, as opposed to what are recommendations for improvement

•The “three level” comment structure is one tool to facilitate this distinction

•Inline edits vs. marginal comments

•Communication•With author, editor, and/or colleague

Tips and Tricks

•“Three paragraph” rule to see if document is ready for full review

•Nominalizations

•Strong and weak sentence parts

•Flipping back and forth-usually not a good sign

•Watch for “buried leads”-important ideas or statements that are hard to find within the document

Tips and Tricks

•For internal reviews, consulting with the author during review may be appropriate and helpful

•Reviewing and revising your own work-”sleep on it”

•Two stage review-read, then either re-read or review your comments

•Reviewing prior comments (yours or those of others) can be helpful

•Don’t repeat comments unnecessarily

Scientific Review Case Study

Review Case Study

•Document: 2012 Surgeon General's Report—Preventing Tobacco Use Among Youth and Young Adults (1395 pages, 7 chapters)

•Type of review: policy•Timeframe for review: 2 weeks•Methods:

•Multiple reviewers due to length of document•Discussion with authors during review process to assure that highest priority comments were addressed•Consolidation and adjudication of all comments

•If multiple reviewers involved, must “speak with one voice”

Resources for Review

•Medline/PubMed (www.ncbi.nlm.nih.gov/pubmed)

•Example use: does a reference really say what the authors assert they say? Often can be assessed by reviewing the abstract

•Clinical Trials registry (www.clinicaltrials.gov)•Most journals require clinical trials to be registered here

•Agency or other medical libraries (and librarians)

Building Experience as a Reviewer

•How do you get to Carnegie Hall, or become a good reviewer?

•Practice, practice, practice

•Volunteer to review abstracts for COA or other scientific meetings•Serve as a peer reviewer for journals, including Public Health Reports•Review colleague’s manuscripts

•Write abstracts, reports, columns, journal articles, articles for COA Frontline etc.

What’s In It for Me?

•Some journals (e.g. Annals of Internal Medicine) offer continuing education (CE) credit to reviewers•Some journals “credit” editors as part of the final publication•In some agencies, scientific review may be a part of leadership responsibilities

•Associate Director for Science, Senior Scientist•Building scientific and public health fund of knowledge, in or outside of your specialty

The Lighter Side of Being a Reviewer

•“Employment is a major cause of morbidity and mortality.”

•“A limitation of this survey is that it is prone to hat misclassification bias.”

•“We blindfolded reviewers…”

Acknowledgments

•Mary Ari

•James Stephens

•Rick Peavy

•LT Neelam Ghiya

Office of the Associate Director for Science