critical appraisal of systematic reviews douglas newberry
TRANSCRIPT
![Page 1: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/1.jpg)
Critical Appraisal of Systematic Reviews
Douglas Newberry
![Page 2: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/2.jpg)
Systematic Reviews —
or How to make a Monkey out of EBM without hardly
trying!
![Page 3: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/3.jpg)
Systematic Reviews:Objectives:
• Appraise a systematic review for validity
• Discuss Meta Analysis / use Odds Ratios
• Obtain Number Needed to Treat (NNT) from Odds Ratios
• Consider clinical implications of a Systematic Review {including when to bin it instead!}
![Page 4: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/4.jpg)
We can see further than our forbearers because we stand on
the shoulders of Giants{and have better spectacles}
• these ideas are cribbed unashamedly from friends, books & previous courses
![Page 5: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/5.jpg)
Systematic Reviews:What are your Objectives:
What do you want to cover?
Please interject with helpful questions!
![Page 6: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/6.jpg)
Did I really want a systematic review?
(but please do not pretend)
• admit your ignorance — expert review or consensus guidelines > broad introduction, cover many areas (class C evidence).
• if the question is important > formulate it!
• Systematic review > narrow but rigorous focus.
![Page 7: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/7.jpg)
Systematic Reviews — Where do I start:
• Start with your 4 (or 3) part clinical question!
• Is a systematic review a sensible approach?
• Does THIS systematic review address MY question?
• Is it a systematic review at all?
![Page 8: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/8.jpg)
Is it a systematic review? does it:• define a four part (answerable) clinical
question?
• combine Randomized Controlled Trials (RCT’s)?
• describe PRE-DEFINED search methods?
• PRE-DEFINED inclusion criteria?
• PRE-DEFINED methodological exclusion criteria?
![Page 9: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/9.jpg)
Sceptical View? Take it with a grain of salt:
• transparent declaration of funding of work?
• Drug Company sponsorship of Reviews vs. Methodological quality>Cochrane review!
• who employs the authors?
• open discussion of existing controversy & commercial gain?
• Don’t waste salt on your food, keep it for your reading!
![Page 10: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/10.jpg)
Meta analysis — combine what with what?
• Low Molecular Weight Heparin (LMWH) in hip surgery — begin before or after the operation?
• meta analysis of placebo controlled RCT’s of heparin in hip surgery >>
• pre-op & post-op LMWH vs. placebo
• post-op LMWH Vs placebo
• pre-operative >> less intra-op bleeding??
![Page 11: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/11.jpg)
Can we believe it ?
• bias free search & inclusion criteria?
• appraisal of methodology of primary studies?
• consistent results from all primary studies?
– if not, are the differences sensibly explained?
• are the conclusions supported by the data?
![Page 12: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/12.jpg)
If we believe it — does it apply to our patient?
• Is our patient (or population) so different from those in the primary studies that the results may not apply?
• consider differences in:
– time — many things change.
– culture — both treatments and values of outcomes can be different
– stage of illness or prevalence can effect results.
![Page 13: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/13.jpg)
We believe it ! but—>> does it matter?
• Is the benefit worthwhile to our patient?
• Ask the patient about cultural values.
• Think about Relative Risk Reduction vs. Absolute Risk to our patient.
• Potential benefit is the Absolute risk avoided in our patient = Absolute Risk Reduction (ARR)!
![Page 14: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/14.jpg)
Absolute Risk—> The risk our patient is facing!
• How likely is our patient to die (or have the outcome of interest) without intervention? = Control Event Rate (CER)
• consider this individual patient’s risk factors to estimate Patient Expected Event Rate = PEER.
• Absolute Risk usually increases with age.
• Improvement measured as Absolute Risk Reduction (ARR)
![Page 15: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/15.jpg)
Relative Risk Reduction:• Usually reported in studies.
• Ratio of the improvement of outcome over outcome without intervention (Rx):
• {Control Event Rate (CER) — Experimental Event Rate (EER)} / CER
• i.e. {CER-EER}/CER
• often independent of prevalence!
• often similar at different ages!
![Page 16: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/16.jpg)
Our patient wants an absolute Risk Reduction (ARR):
• is a 40% reduction in Cardiac Risk worth taking pills daily for 10 years?? >vote!
• if I have a 30% risk of MI or death {30 out of 100 people like me will suffer MI or death} in next 10 years > 40% RRR >> only 18 out of 100 will have MI or death. ARR = 12 out of 100! >>I like that!
• BUT if I have a 1% risk in 10 years, 40% less is a 0.6% risk >> hardly different!
![Page 17: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/17.jpg)
Number Needed to Treat (NNT) (very trendy but tricky):
• only defined for specific prevalence-Patient’s Expected Event Rate=PEER!
• only defined for a specific intervention!• only defined for a specific outcome!
– eg. Pravastatin™ 40 mg nocte x10 years, in a 65 year old male, ex-smoker with high BP and Diabetes, to reduce MI or Death.
• NNT is the inverse of Absolute Risk Reduction: i.e. NNT = 1/ARR
![Page 18: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/18.jpg)
Number Needed to Treat (NNT) for previous example:
• 12 fewer MI or death in 10 years per 100 persons treated: ARR=12/100
• NNT = 1/(12/100)=100/12= 8.3
• But the same Relative Risk Reduction (RRR) of 40% with a low prevalence:
• 0.4 fewer MI/death per 100 treated, ARR=0.4/100.
• NNT = 1/(0.4/100) = 100/0.4 = 250!
![Page 19: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/19.jpg)
Why Odds Ratios? > compare results of different studies.
• consider 2x2 table:
• RRR is (a-b/a) — but you can only go in rows within same study!
• Odds ratio is (a/c)/(b/d) = ad / bc — the individual ratios are in columns, and therefore are independent of the prevalence which is different in different studies.
• must use odds ratios to combine RCT’s
![Page 20: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/20.jpg)
Odds Ratio (OR) to NNT — is the improvement worth the trouble?
• 1>OR>0, lower the OR = better the treatment (Rx) >> lower NNT.
• for any OR, NNT is lowest when PEER=0.5
• estimate the PEER (patient’s risk)
• apply the OR to get patient's NNT.
![Page 21: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/21.jpg)
Convert PEER & OR to NNT:
Odds Ratio (OR)Control CER 0.9 0.7 0.5
Event 0.1 110 36 21Rate(CER) 0.5 38 11 6{applyPEER 0.9 101 27 12here}
![Page 22: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/22.jpg)
Formula used in the table:
NNT=1- {PEER * (1-OR)}
(1-PEER)*(PEER)*(1-OR)
![Page 23: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/23.jpg)
Table induced nausea!• lower OR >> lower NNT
• Patient needs to be at risk (non-trivial PEER) in order for risk reduction to be worth the effort.
• for any OR, NNT lowest when PEER=0.5
• more effective treatment > lower NNT
• BUT are your patient’s values satisfied by the intervention and its sequelae?
![Page 24: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/24.jpg)
Subgroup analysis: Sceptical unless:• the subgroups make biological and clinical
sense?
• the differences are both clinically & statistically significant?
• was a-priori hypothesis (before this data)?
• other evidence supports these sub-groups?
• few (OK) or many (nix) sub-group analyses?
![Page 25: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/25.jpg)
Any Questions?
![Page 26: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/26.jpg)
Summary 1: Set your goals.• define your 4 (or 3) part question.
• do you want a true systematic review?
• does this narrow review address my question?
• PRE-DEFINED search, inclusion, exclusion!
![Page 27: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/27.jpg)
Summary 2: Be Sceptical!• look for bias, conflict of interest.
• critical appraisal of primary studies?
• consistent results? if not, why not?
• does our patient fit the groups studied?
• does it matter to our patient?
![Page 28: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/28.jpg)
Summary 3: Risks that matter.• Absolute risk > estimate the Patient
Expected Event Rate (PEER)
• obtain Relative Risk Reduction (RRR) or Odds Ratio (OR) from a Meta-analysis
• plug into a table to estimate Number Needed to Treat (NNT)
![Page 29: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/29.jpg)
Summary 4: Sceptical & common sense!
• beware of post-hoc sub-group analysis, especially if multiple.
• step back and consider if the systematic review really related to our patient’s situation (PEER), culture and expectations?
• do not loose sight of common sense!
![Page 30: Critical Appraisal of Systematic Reviews Douglas Newberry](https://reader037.vdocuments.mx/reader037/viewer/2022110322/56649d215503460f949f73a0/html5/thumbnails/30.jpg)
Coffee Now!
• Small Groups Afterwards