Critical appraisal:Systematic Reviews and

Clinical Practice Guidelines forDrug Therapy

Nancy J. Lee, PharmD, BCPSResearch fellow, Drug Effectiveness Review Project

Oregon Evidence-based Practice CenterOregon Health and Science University

To receive 1.25 AMA PRA Category 1 Credits™ you must review this section and answer CME questions at the end.

Release date: January 2009 Expiration date: January 2012

Attachments

• The attachments tab in the upper right hand corner contains documents that supplement the presentation

• Handouts of slides and a glossary of terms can be found under this tab and are available to print out for your use

• URL to online resources are also available

Program funding

This work was made possible by a grant from the state Attorney General Consumer and Prescriber Education Program which is funded by the multi-

state settlement of consumer fraud claims regarding the marketing of the prescription drug Neurontin®.

Continuing education sponsors:

The following activity is jointly sponsored by:The University of Texas Southwestern Medical Center

and the Federation of State Medical Board’s Research and Education Foundation.

Program Speaker/Author: Nancy J. Lee, PharmD, BCPS Research fellow, Oregon Health and Science University, Oregon Evidence-base Practice Center, Drug Effectiveness Review Project

Course Director: Barbara S. Schneidman, MD, MPH Federation of State Medical Boards Research and Education Foundation, Secretary Federation of State Medical Boards, Interim President and Chief Executive Officer

Program Directors: David Pass, MD Director, Health Resources Commission, Oregon Office for Health Policy and Research Dean Haxby, PharmD Associate Professor of Pharmacy Practice, Oregon State University College of Pharmacy Daniel Hartung, PharmD, MPH Assistant Professor of Pharmacy Practice, Oregon State University College of Pharmacy

Target Audience: This educational activity is intended for those that are involved with committees involved with medication use policies and for health care professionals who are involved with medication prescribing.

Educational Objectives: Upon completion of this activity, participants should be able to: recognize benefits and limitations of systematic reviews and clinical practice guidelines; assess quality of systematic reviews and clinical practice guidelines; identify differences between systematic reviews, narrative reviews, and meta-analyses; recognize components of forest plots used in meta-analyses in systematic reviews; review the grading of the strength of evidence used in clinical practice guideline development.

CME information

CME policiesAccreditation: This activity has been planned and implemented in accordance with the Essential Areas & Policies of the Accreditation Council for Continuing Medical Education through the joint sponsorship of The University of Texas Southwestern Medical Center and the Federation of State Medical Boards Research and Education Foundation. The University of Texas Southwestern Medical Center is accredited by the ACCME to provide continuing medical education for physicians.

Credit Designation: The University of Texas Southwestern Medical Center designates this educational activity for a maximum of 1.25 AMA PRA Category 1 Credits™. Physicians should only claim credit commensurate with the extent of their participation in the activity.

Conflict of Interest: It is the policy of UT Southwestern Medical Center that participants in CME activities should be made aware of any affiliation or financial interest that may affect the author’s presentation. Each author has completed and signed a conflict of interest statement. The faculty members’ relationships will be disclosed in the course material.

Discussion of Off-Label Use: Because this course is meant to educate physicians with what is currently in use and what may be available in the future, “off-label” use may be discussed. Authors have been requested to inform the audience when off-label use is discussed.

DISCLOSURE TO PARTICIPANTS

It is the policy of the CME Office at The University of Texas Southwestern Medical Center to ensure balance, independence, objectivity, and scientific rigor in all directly or jointly sponsored educational activities. Program directors and speakers have completed and signed a conflict of interest statement disclosing a financial or other relationship with a commercial interest related directly or indirectly to the program.

Information and opinion offered by the speakers represent their viewpoints. Conclusions drawn by the audience should be derived from careful consideration of all available scientific information. Products may be discussed in treatment outside current approved labeling.

FINANCIAL RELATIONSHIP DISCLOSURE

Faculty Type of Relationship/Name of Commercial Interest(s)

David Pass, MD NoneDean Haxby, PharmD Employment/CareOregonDaniel Hartung, PharmD, MPH NoneNancy Lee, PharmD, BCPS NoneBarbara S. Schneidman, MD, MPH None

Learning objectivesI. Systematic reviews

– Recognize benefits and limitations– Assess quality of systematic reviews– Identify the differences between systematic reviews,

narrative reviews, and meta-analyses– Recognize components of forest and funnel plots used

in systematic reviews with meta-analysesII. Guidelines

– Identify strengths and weaknesses– Assess and recognize quality components– Review grading of the strength of evidence used in

guidelines

I. Systematic Reviews:Outline

• Why, When, What?– Benefits and limitations

• Steps in conducting Systematic Reviews– Scientific process

• Quality assessment of Systematic Reviews– Tools and checklists

Why are systematic reviews needed?• Too much information• Not enough time

– More than 2 million articles published yearly from more than 200 biomedical journals

– Results can often be contradicted by subsequent trials

• Taken together, a clearer picture can emerge– Minimize biases– Increase statistical power– Improve generalizability– Improve allocation of resources for

other needed trials = minimize funding of unnecessary trials

Fergusson D, et al. Clin Trials 2005; 2:218-32

Did trialists review all the literature before conducting their own study?

After RCT #12, the cumulative effect estimate (OR) stabilizes in the range of 0.25 – 0.35.

Throughout the cumulative meta-analysis, the upper limit of the confidence interval never crossed 0.65

The largest trial published in 1992 was referenced in 7 of 44 (16%) of trials published

more than 1 year later.

Overall, ~20% of trials cited previous trials in their study.

69 trials

When are systematic reviews needed?

• When an important question needs to be addressed– Gaps in the literature or conflicting results

• When there is uncertainty regarding an intervention– Uncertainty may lie in:

• Population, Intervention, Outcomes

• When several primary studies exist– Lack of strong evidence

Limitations of systematic reviews• Only as good as what is available

and what is included– Issue of publication bias

• Restricted to published results– Quality of individual trials

• “Garbage In, Garbage Out”

• Good quality systematic reviews typically do not address all the issues relevant for decision making– Evidence outside the scope of the

review may be relevant and needed for decision making

– Cost and implementation implications may not always be addressed

Limitations of systematic reviews

• Unrealistic expectations– What if results conflict with a good quality large landmark trial?– About 10-23% of large trials disagreed with meta-analyses*

• May not always include the most up to date studies– When was the last literature search conducted?– Estimate: 3-5 years**

• Does not make decisions for the user– These are not guidelines– The reader uses their own judgment

*Ioannidis, et al. JAMA 1998; 279:1089-93. **Shojoania, et al. Ann Intern Med 2007; 147:224-33.

What it is and isn’tFeature Narrative review (traditional) Systematic review

Questions Often broad in scope Focused clinical question(s)

Sources and search strategy

Not usually specified; potentially biased

Comprehensive and explicit search strategy

Study eligibility Not usually specified; potentially biased

Prespecified; criterion-based; uniformly applied

Appraisal Variable; assessment of the quality of evidence typically

not reported

Rigorous critical appraisal; typically includes quality assessment of evidence and provides insight into

potential study biases

Synthesis Often qualitative Qualitative with or without meta-analyses

Inferences Sometimes evidence based Usually evidence based

Adapted from Cook DJ, et al. Ann Intern Med 1997; 126:376-80.

The advantage of using carefully done, systematic reviews becomes clear when we observe how often mistakes are made when

research is reviewed non-systematically, whether by experts or others. The costs of

mistaken conclusions based on non-systematic reviews can be high.

-Oxman, AD

I. Systematic Reviews:Outline

• Why, When, What?– Benefits and limitations

• Steps in conducting Systematic Reviews– Scientific process

• Quality assessment of Systematic Reviews– Tools and checklists

Systematic Reviews: A scientific process

Figure 1. Copyright ©1997 BMJ Publishing Group Ltd. from Greenhalgh T. BMJ 1997;315:672-5.

• Developed a priori– Most important– Relevant and sensible to practitioners and patients?– Typically not changed during the review process

• What are we asking?– Efficacy– Effectiveness

• Well-defined?– PICOS

• Any exclusions?– Language restrictions or type of study design

What’s the purpose and question?

What was the study eligibility?• Determines what studies get included in a

systematic review– Formed a priori– Applied uniformly by at least 2 reviewers (dual review)

• Study inclusion and exclusion criteria should relate to the areas defined by PICO(S)– Population– Intervention– Comparator– Outcome– Setting/study design

Study eligibility• What are the consequences of being too inclusive

or exclusive?

– Too inclusive• Scope is too large• Lose focus of question• Main point may be lost• May be difficult to interpret

– Too exclusive• Scope is too narrow• Potential to exclude important trials• May end up not having enough evidence• If unaware, could lead to biased conclusions

Example: Study eligibilityPopulation adults and children with type 2 diabetes mellitus Intervention, comparator

sitagliptin; placebo; other oral antihyperglycemic agents

Outcomes all cause mortality, micro-and macrovascular disease, quality of life (Intermediate outcomes: A1c)

Study design For efficacy/effectiveness: RCTs and good quality systematic reviewsFor harms: RCTs, good systematic reviews, large comparative cohort observational studies

Study duration ≥ 12 weeks in durationExclusions poor quality trials/studies were excluded from

analyses

Finding all relevant studies:Search strategy

• Medical librarian important

• Key search terms should at the very least be reported

• Were any significant studies missing?– If yes, why?

Was the search strategy

comprehensive?

Example: Search strategy

Finding all relevant studies: Sources • Electronic databases

– MEDLINE (Ovid/PubMed)– Cochrane Library– EMBASE– PsychINFO– CINAHL

• Hand searching– Reference lists of trials and/or reviews– Journals

• Sources for unpublished information– FDA website– Clinical Trials.gov– Registeries

• Industry dossiers

Selection of studies

• Review titles and abstracts from initial search• Review of full text articles• Uniform application of study eligibility criteria• Dual review for each step

– Disagreements resolved by consensus

Was the selection of studies unbiased?

Issue of publication bias

Adapted from Cochrane Open Learning. Module 15. Publication bias 2002.

• “Positive” studies are more likely to be published…− Rapidly, in English, more than once

• Failure to publish or submit “negative” studies by investigators, peer reviewers, editors and Pharma− May knowingly or unknowingly

influence the results toward the positive

Was it addressed?

Scargle. J of Scientific Explor 2000; 14(1):91-106.

“researchers and statisticians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried

out…. The extreme view of this problem, the “file drawer problem,” is that the journals are filled with the 5% of the

studies that show Type I errors, while the file drawers back at the lab are filled with the 95% of the studies that

show nonsignificant (e.g., p > .05) results.”

(Rosenthal, 1979, p. 638)

Investigating for presence of publication bias

• Visually check for asymmetry in funnel plots– NOT a tool to “diagnose” bias

• Potential sources of asymmetry– True heterogeneity– Data irregularities– Chance

• Other statistical methods– Ask a biostatistician

Egger, et al. BMJ 1997; 315:629-34. Figure 1 from Peters, et al. JAMA 2006; 295:676-80.

Ways to minimize publication bias in the review process

• Identify duplicate publications

• Contact study authors or manufacturer– Often difficult to obtain information– Time intensive

• Check sources for grey literature– FDA review documents– Clinical trial registries– Databases

• Check for any language restrictionsRising, et al. PLoS Med 5(11):e217.

Quality assessment of included studies

• >25 different tools– Jadad scale, Risk of Bias tool, DERP method (for trials)– Other scales or checklists (for observational studies)

• How were poor-or low quality trials handled in the review?– Were these excluded?– Sensitivity analyses?

Was quality assessment of individual studies conducted and reported in the systematic review?

Example

Bjelakovic, et al. Lancet 2004; 364:1219-28.

Data abstraction• Dual abstraction and review

• Types of data abstracted:– Study design– Setting– Population characteristics (age, sex, ethnicity)– Inclusion/exclusion criteria– Interventions– Comparisons– Number screened, eligible, enrolled– Number withdrawn– Method of outcome ascertainment– Results– Adverse events

Data synthesis

• Two methods: qualitative and quantitative

• Qualitative– Discussion of results (synthesis)

• in relation to each other• in relation to study quality• Not a reporting of results from each study

Adapted from Cochrane Collaboration open learning materials for reviewers 2002-2003.

Data synthesis• Quantitative or meta-analyses

– Statistical method for combining results from >1 study• Advantage: provides an estimate of treatment effect• Disadvantage: misleading estimate if used inappropriately

– Misuse of terminology• Systematic review and Meta-analysis = NOT

the same

SR MA

Adapted from Cochrane Collaboration open learning materials for reviewers 2002-2003.

Meta-analysis

• The review should provide enough information about the included studies for you to judge whether combining results was appropriate.

• Two types of heterogeneity– Clinical heterogeneity

• Does it make clinical sense to combine these studies?– Statistical heterogeneity

• Are there inconsistencies in the results?• Calculation of Q-or I-squared statistic

• Common sources of heterogeneity– Clinical diversity between studies, conflicts of interest, and

differences in study quality

Is combining results of individual studies appropriate?

Adapted from Cochrane Collaboration open learning materials for reviewers 2002-2003.

Example: Clinical heterogeneity?

How to read a Forest plot

Trials

Trials

Line of no effect

Each square box= point estimate

Diamond= pooled estimate of trials

Diamond= pooled estimate of trials

Size of the square= proportional to weight/size of the study precision

Horizontal line= confidence interval

Forest plot adapted from Bjelakovic, et al. Lancet 2004; 364:1219-28.

• Two common methods– Fixed effects model

• Assumes homogeneity

– Random effects model• Assumes heterogeneity

– Use both methods and select 1 to present• Should briefly discuss why a certain method was

selected

What statistical method was used for the meta-analysis?

Invalid methods of synthesis

• Picking and choosing– Pick what you like, ignore what you don’t like

• Searching for proof– Data dredging or data mining

• Vote counting– Counting the number of studies with positive and

negative results without considering study quality

Bridging the resultsto the conclusion

• Do conclusions reflect the uncertainty in the evidence?

• Are gaps identified and recommendations for future research provided?

I. Systematic Reviews:Outline

• Why, When, What?– Benefits and limitations

• Steps in conducting Systematic Reviews– Scientific process

• Quality assessment of Systematic Reviews– Tools and checklists

Key questions to ask when assessing quality of systematic reviews

• Is there a clear, focused, clinically relevant question?

• Were study eligibility criteria reported and rationale provided (if needed)?

• Was the search for relevant studies detailed and exhaustive?

• Were included trials assessed for quality and were the assessments reproducible?

• How was data synthesized and was this appropriate?

• Are the conclusion statements clear and reflect the results from the evidence that was reviewed?

Tools and lists for assessing systematic review quality

• >10 different scales and checklists– Oxman and Guyatt– Sacks, et al– DERP method

Oxman and Guyatt

1. Were the search methods used to find evidence on the key questions?

2. Was the search for evidence reasonably comprehensive?3. Were the criteria used for deciding which studies to include

reported? 4. Was bias in the selection of studies avoided? 5. Were the criteria used for assessing the validity of the

included studies reported? 6. Was the validity of all the studies referred to in the text

assessed using appropriate criteria? 7. Were the methods used to combine the findings of the

relevant studies reported? 8. Were the findings of the relevant studies combined

appropriately? 9. Were the conclusions made by the author(s) supported by the

data reported? 10. How would you rate the scientific quality of this overview?

Shea B, et al. Eval Health Prof 2002; 25(1):116-29.

Using Oxman and Guyatt method

Author

From DERP report. http://www.ohsu.edu/drugeffectiveness

Using Oxman and Guyatt method(continued)

Sacks, et al1. Prospective design

a. Protocol b. Literature search c. Lists of trials analyzed d. Log of rejected trials e. Treatment assignment f. Ranges of patients g. Ranges of treatment h. Ranges of diagnosis

2. Combinabilitya. Criteria b. Measurement

3. Control of biasa. Selection bias b. Data-extraction biasc. Interobserver agreement d. Source of support

4. Statistical analysisa. Statistical methods b. Statistical errors c. Confidence intervals d. Subgroup analysis

5. Sensitivity analysisa. Quality assessment b. Varying methods c. Publication bias

6. Application of resultsa. Caveats b. Economic impact

7. Language

DERP method

50From DERP report. http://www.ohsu.edu/drugeffectiveness

Author

Author

Database of Abstracts of Reviews of Effects

Author X

Summary: Systematic Reviews

• Advantages and disadvantages– Can minimize biases that exist in individual studies– May not answer all questions of interest

• Systematic reviews and meta-analyses are not synonymous– Meta-analysis is a statistical method of combining

studies

• Each step of the process should be questioned– Comprehensive search of evidence– Quality assessment of individual trials– Appropriate method of synthesis

Appraisal of guidelines

II. Guidelines: Outline

• What is the purpose and what are the potential benefits and limitations?

• Why do we need to critically assess guidelines?

• Quality assessment of guidelines

• Tools to help evaluate guidelines

Guidelines: steps beyond a review• Incorporates the judgments and values involved in

making recommendations• Addresses larger spectrum of issues relevant for clinical

decision making

• Purpose:– Provide clinical practice recommendations– Improve quality of care and outcomes– Seek to influence change in clinical practice– Reduce inappropriate variation in practice– Shed light on gaps in the evidence RCTsRCTs

SRsSRs

Guidelines are not intended to…

• Provide a black and white answer for complex situations

• Substitute for clinical insight and judgment• Be a legal resource in malpractice cases• Prompt providers to withdraw availability or

coverage of therapies• Hinder or discourage scientific progress

Woolf S, et al. BMJ 1999; 318:527-30.

Why is it necessary tocritically assess clinical practice guidelines?

• There are > 2,500 published guidelines – Multiple guidelines with differing recommendations– Not all guidelines are of good/high quality– Consensus-based– “Evidence-based” (systematic methods, transparent)

• Many “stakeholders” who are invested in the influence of their guidelines– Government organizations and healthcare systems– Professional societies– Pharmaceutical industry

A glance at guidelines from 1988-1998

• 3 items assessed1) Description of

professionals involved

2) Search undertaken3) Explicit grading of

evidence for recommendation

N= 431, guidelines assessed

Grilli, et al. Lancet 2000; 355:103-6.

Results from Grilli, et al

All 3 criteria were met only in 5% of the identified guidelines and 54% did not meet ANY of the items.

Assessing quality

• Who were involved in the decision making process?– Were all relevant perspectives considered?– To what extent were the funders of the guideline

involved in the process?– Conflicts of interests declared for each participant?

• Were all important practice options and clinically relevant outcomes considered?– What was excluded and was rationale provided?

• How were the relative values of the outcomes weighed in terms of importance?

11 pain specialists involved—over 2 days in New Orleans

Target audience: primary care physicians, internal medicine physicians, geriatric physicians, and psychiatrists treating chronic pain.

Conflict of interest reported for 10/11 members

8 of 10 members received some sort of funding from Eli Lilly who provided an educational grant for this guideline

Consensus guidelines: Assessment, diagnosis, and treatment of diabetic peripheral neuropathic pain. Mayo Clinic Proceedings 2006; 81(4):S1-36.

Assessing quality

• How was evidence retrieved?– Was it comprehensive?

• Was there explicit description of how “evidence” was used?– Systematic reviews?– Was there an approach to the hierarchy of evidence?

• Was quality of the evidence assessed and reported?

• How was the body of evidence graded?

Chou, et al. Ann Intern Med 2007; 147:505-14.

• -ing the strength of the evidence and recommendations reported in guidelines– To provide a systematic and explicit approach to

making judgments involved in a guideline process that can be used by all guideline developers

One method ofassessing the body of evidence

The approach considers:

• Strength of the body of evidence– Study design– Risk of bias or

limitations– Consistency of results– Precision– Directness of evidence

• Strength of recommendation:– Strong vs. Weak

Example: GRADE table

The AGREE instrument

Summary: Guidelines• Incorporates values and judgments

• Can improve care by reducing variation in practice– Not meant to provide black and white answers for complex

problems

• Not all guidelines are the same– Consensus-based approach– Evidence-informed approach

• Important to question each step of the process– Who was involved?– How was evidence retrieved, synthesized, and graded?– How were recommendation decisions made?

Acknowledgements• Attorney General Consumer and Prescriber

Education Program

• Members of the technical advisory committee of this grant

• Office for Oregon Health Policy and Research

• The University of Texas Southwestern Medical Center

• The Federation of State Medical Board’s Research and Education Foundation

CME instructions

• Please complete the survey, CME questions, and program evaluation after this slide

• Don’t forget to click the finish button at the end of the CME questions

• You should be directly linked to a CME form which you will need to fill out and fax, email, or mail in order to receive credit hours

PROPERTIES

On passing, 'Finish' button: Goes to URLOn failing, 'Finish' button: Goes to Next SlideAllow user to leave quiz: After user has completed quizUser may view slides after quiz: At any timeUser may attempt quiz: Unlimited times