systematic reviews & meta-analysis - duke university€¦ · pubmed clinical queries:...
TRANSCRIPT
Systematic Reviews &Meta-analysis
Clay BordleyDepartment of Pediatrics
Division of Hospital & Emergency Medicine
Jane P. GagliardiDepartment of Psychiatry and Behavioral Sciences
Department of Medicine
First a Case….
3 mo w/ sneezin’, wheezin’ and fever
• Congested for 4 days, fast breathing for past 24 hours– Breast and bottle fed, but having trouble sucking and breathing– Decreased wet diapers
• 3yr cousin with URI sx’s• PMHx: 39 wk, unremarkable P/L/D• SHx: 16yr mother (sucking her thumb), lives with MGM, mo and cousins• PE
– 38.0 186 64, 93% sat– Alert, obvious congestion and increased WOB’ing, nl TM’s, (+) retractions, defuse
wheezing, no murmur, nl CR and pulses, abd soft, no HSM
• Your impression: – Bronchiolitis, fever, borderline sats
What Clinical Questions Do You Have?
Disclosures
W. Clayton Bordley, MD, MPH
Today’s Clinical Question
Is hypertonic saline effective in infants with bronchiolitis?
Exercise #1
Search in groups of 2
Time: 3-5 minutes
Be prepared to report to group
1. How did you do your search2. What source did you choose?
3. Did you get an answer?
What Did You Find?
Is This a Systematic Review?
eMedicne
Did You Use PICOTT?
• Patient• Intervention• Control• Outcome• Type of question• Type of study
• Infants w/ bronchiolitis• Hypertonic saline• None• Effectiveness (?)• ?• ?
Exercise #2
• Connie searched and came up with 7 candidate studies
Exercise #2• In the next FIVE MINUTES:
– Critically appraise your article• Follow-up• Randomization• Intention to treat• Similar at baseline• Blinding• Equal treatment outside the intervention
• Summarize the methods and results• Be ready to share your answers out loud
• 5 minutes to CAT the articles• Follow-up• Randomization• Intention to treat• Similar at baseline• Blinding• Equal treatment outside the intervention
What overall grade would you assign your study?
1 2 3 4 5
0% 0% 0%0%0%
1. A (excellent, few sources of bias)2. B (not bad, despite some bias)3. C (mediocre, some obvious bias)4. D (barely makes the cut, lots of
bias)5. F (may not even be worth
reading)
Please remember the grade you assigned(will return to this a bit later)
Learning Objectives
• Understand the distinction between reviews, systematic reviews and meta-analysis
• Understand the basic methods & outcomes reported in meta-analysis
• Interpret a forest plot• Understand the critical appraisal elements that
apply to systematic reviews• Understand role systematic of reviews in
guidelines and other publication types• Use a systematic review to help guide clinical
decision-making
Components of SystematicReviews
• Specific objective(s)• Clearly described, reproducible methods
– Databases and search terms• A priori inclusion / exclusion criteria• Data extracted by more than one reviewer• Consideration of quality• Evidence table(s)• Results
– Qualitative and/or quantitative
Specific Objective(s)
• A focused clinical question• (May look suspiciously like a PICOTT)
Clearly Described Methods
• LITERATURE SEARCH:
– Search Terms– Database(s)– Time Frame
Inclusion/Exclusion Criteria
• This needs to be specified A PRIORI:– What types of studies will be included?– What characteristics of studies will be
accepted?– Is there any consideration of QUALITY of the
studies to be included?
Consideration of quality - Assessments of study quality should resemble:
1 2 3 4 5
0% 0% 0%0%0%
1. PICOTT2. Validity Criteria3. Cochrane review4. Performance
Improvement5. Systematic review
Data Abstraction
• How many reviewers?• Should the reviewer(s) be aware of the
study hypothesis?• What sort of information should the
reviewer(s) abstract?• How are conflicts resolved?
Results
• Flow Diagram for Study Selection• Evidence Tables• Forest Plots
Impact of Diet Soda on Impulsivity
Favors regular Favors diet
*adapted from article addressing different question
Summarizing the Evidence
BREAK!
Which of the following is NOT a necessary part of a systematic review?
a) b) c) d) e)
0% 0% 0%0%0%
a) Focused clinical questionb) Reproducible search
methodsc) Inclusion/exclusion criteriad) Statistical analysise) Consideration of study
quality
Meta-analysis
Meta-analysis
(definition): mathematical synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way
The Basic Principles• Each study reports a measure of effect
– Rate, RR, OR, risk difference, others• Summary (pooled) effect measure is calculated
– weighted average of the individual studies• Weighting may be based on sample size (usually) and/or study quality
• Confidence intervals – Communicates the precision of the summary estimate, – Allows calculation of a P value
• Measure of variability among results (aka heterogeneity)– Is the variation compatible with random variation?– Is the variation sufficiently large to suggest the studies assessed
different “truths” in their results? – Note: estimate of whether or not methods are similar enough to
combine should have been done while compiling evidence table
Review, Systematic Review and Meta-analysis
Exercise #3
• Who is familiar with Forest Plots?• Create a Forest Plot
– Based on the 7 RCT’s you reviewed– Each group come down and add their results
to the Forest plot
Favors treatment Favors control
What is the outcome measure?
What is the unit of measure?
Let’s Look at Some Prettier Examples
Forest Plots• The label explains comparison and outcome
Forest Plots• The data for each trial are represented, divided into
experimental and control groups• The calculated effect measure is presented numerically• The % weight given to each study is reported
Forest Plots• The result of each study is plotted graphically with a block• The size of the block for each study is proportional to the % weight. • The horizontal line represents the confidence interval.
• The label above the graph provides information about statistics• The horizontal line at the bottom depicts the scale for the treatment
effect
Forest Plots
Forest Plots• The vertical line in the middle is where treatment and control have
the same effect – THE LINE OF NO DIFFERENCE
Forest Plots• The pooled estimate is given as a diamond shape
– The horizontal width of the diamond is the confidence interval of the pooled estimate
• The pooled estimate is also given numerically
Forest Plots• Heterogeneity: The forest plot displays the results of the Chi square
test and the p value
Heterogeneity
• Inevitably, studies brought together in a systematic review will differ.
• The amount of variability in the results from studies in a systematic review is termed heterogeneity.
• What are some possible sources of heterogeneity?
Measures of Heterogeneity
• Eyeball test• Chi Square• I2 Inconsistency
“eyeball test”
Bordley 2013
Measures of HeterogeneityEyeball test
• Chi Square– Assesses whether observed differences in results
are compatible with chance alone. – A low P value (or a large chi-squared statistic
relative to its degree of freedom) provides evidence of heterogeneity (variation in effect estimates beyond chance).
• I2 Inconsistency– Moves the focus away from testing whether
heterogeneity is present to assessing its impact on the meta-analysis.
Back to Our Case
Validity Criteria for a Systematic Review
• Focused clinical question?• Appropriate study inclusion criteria?• Thorough search?
– unlikely to miss relevant studies?• Quality of included studies assessed? • Heterogeneity?• Sufficient follow-up (duration, subjects
accounted for)? • Generalizable to your clinical scenario?
It All Comes Back to the Search
Assessing Quality of Included Studies
• Why– Garbage in leads to garbage out– Individual study bias affects meta-analysis
result• How: Quality Scores
– 25 scales & 9 checklists (e.g., Jadad)– Cochrane method:
• 7 domains and risk of bias for each– Sensitivity analysis
Exercise #4
APPRAISE THE SYSTEMATIC REVIEW• Focused clinical question?• Appropriate study inclusion criteria?• Thorough search?
– unlikely to miss relevant studies?• Quality of included studies assessed? • Heterogeneity?• Sufficient follow-up (duration, subjects
accounted for)? • Generalizable to your clinical scenario?
What grade would you assign this systematic review?
1 2 3 4 5
0% 0% 0%0%0%
1. A- nearly perfect minimal bias
2. B-not too bad; some bias3. C-mediocre; may have
biased results4. D-garbage in…5. F-garbage out…
3% saline versus 0.9% saline: Length of hospital stay (days)
Figure 1. 3% saline versus 0.9% saline: Length of hospital stay (days)
3% saline versus 0.9% saline: Rate of Hospitalization
Given the information you have, would you give the baby nebulized saline?
1 2 3 4 5
0% 0% 0%0%0%
1. No, just use humidified air (tell the kid's parents to keep the hot shower running in a closed bathroom)
2. No, just use the nebulized Xopenex
3. Yes, use 0.9% nebulized saline4. Yes, use 3% nebulized saline5. I prefer to conduct my own
clinical trial
Summary Points• All reviews are not Systematic Reviews• Systematic review use explicit and reproducible
methods to answer a focused question• A meta-analysis is a mathematical synthesis of
the results of two or more primary studies that addressed the same hypothesis in the same way
• Meta-analysis increases the precision of a result– Decision to perform involves judgment and
quantitative assessment
Hierarchy of Evidence
Extra stuff
• Systematic reviews are the foundation of other evidence based tools
• Finding systematic reviews
Other Applications of Systematic Reviews
• Practice Guidelines– Combine quality of evidence with strength of
recommendation• Cost-benefit analyses (to derive ranges
used in models• Decision analysis (to derive probabilities
used in models)
Practice Guidelines
Other Studies Built Upon Systematic Reviews
Other Systematic Reviews“The Rational Clinical Examination”
Locate systematic reviews using• Cochrane Database of Systematic Reviews (CDSR)
The CDSR is a full-text database containing systematic reviews and protocols (reviews still in progress) of the effects of health care interventions; mainly randomized controlled trials. "Gold Standard" for high quality, systematic reviews.
• Database of Abstracts of Reviews of Effects (DARE)Structured abstracts of good quality systematic reviews from around the world. Complements CDSR by offering a selection of quality assessed reviews in subjects with no Cochrane review. Also part of the Cochrane Library.
• PubMed Systematic ReviewsPubMed Clinical Queries: Systematic Reviews combines your search term(s) with citations identified as systematic reviews, meta-analyses, reviews of clinical trials, evidence-based medicine, consensus development conferences, and guidelines. No evaluation of comparative quality of different reviews on a topic.
• PubMed | Search TipsHint: you can limit to Publication Type: Meta-Analysis
• CINAHL Plus | Search TipsHint: you can limit with Publication Types: Systematic Review
Effective Ways to Keep Up
Increasing Utility of Systematic Reviews
• Cochrane—standardizing methodology• The PRISMA statement
– reporting systematic reviews and meta-analyses of healthcare interventions
• Critical appraisal criteria – JAMA Users Guide to the Medical Literature
• DARE and PubMed filters – Finding systematic reviews can be easy!
Publishing Systematic Reviews
The CONSORT equivalent