systematic reviews pg 2009

8/3/2019 Systematic Reviews Pg 2009

http://slidepdf.com/reader/full/systematic-reviews-pg-2009 1/63

How to read a Systematic Review:

The FAST tool

Paul GlasziouCentre for Evidence Based Medicine

University of Oxford

www.cebm.net

Find

Appraise

Synthesise

Transferability



The best evidence for

different types of question

Level Treatment Prognosis Diagnosis

I SystematicReview of

SystematicReview of

SystematicReview of

II Randomised

trial

Inception

Cohort

Cross

sectionalIII



Are RCTs always needed for

treatment questions?

Some immediate & dramaticeffects dont need RCTs*

Example:

Child with nasal foreign body

Dislodged with Parent Kiss method

Case series of success 15/19

o Botma J Laryngol Otol 2000

* Glasziou, Chalmers, Rawlins, McCulloch BMJ 2007



How to read a Systematic Review:

The FAST tool

Paul GlasziouCentre for Evidence Based Medicine

University of Oxford

www.cebm.net

Find

Appraise

Synthesise

Transferability



ou are here because?

Y ou want to use systematic reviews forclinical problems

Y ou are helping with a systematic review

Y ou are leading a systematic review

Y our boss told you to attend



Today·s Program

Lecture: Reading systematic reviews Tea break

Small Group: appraisal of a review Lunch & Photograph

Searching for Reviews

Tea break Lecture: Interpreting Reviews

Small Group: Reading Reviews



What do you do?

For an acutely ill patient, you do a search

Y ou find 17 studies:

some show significant results; most do not



The 17 studies : Forest Plot/Blobbogram

A. Which is the smallest study?

B. Which is the largest study?

C. How many are statisticallysignificant?

D. Which studies are large enough?



The 17 studies: of streptokinase for MI



Study RR (95% CI)

0.23 (0.03,1.75)Fletcher 0.57 (0.20,1.66)Dewar

1.35 (0.74,2.45)1st European1.22 (0.67,2.24)Heikinheimo

1.01 (0.55,1.85)Italian

0.70 (0.53,0.92)2nd European0.46 (0.25,0.83)2nd Frankfurt

0.78 (0.48,1.27)1st Australian

2.38 (0.65,8.71)NHLBI SMIT 1.05 (0.48,2.28)Valere

0.96 (0.33,2.80)Frank

0.90 (0.63,1.28)UK Collab2.57 (0.34,19.48)Klein

0.61 (0.42,0.89) Austrian

0.28 (0.03,2.34)Lasierra1.16 (0.84,1.60)N German

0.81 (0.26,2.51)Witchitz

0.85 (0.54,1.34)2nd Australian0.51 (0.33,0.78)3rd European

0.88 (0.62,1.25)ISAM

0.83 (0.75,0.91)GISSI-10.77 (0.70,0.84)ISIS-2

0.80 (0.75,0.85)Overall (95% CI)

Risk ratio0.1 1 10

Streptokinase for Myocardial Infarction



Streptokinase for myocardial infarction



How large should the study be?



What sample size is needed?

For E.H.D.* the usual mortality rate is 0%

What sample size is needed to detect a

reduction in mortality? 100

1,000

100,000 1,000,000

* Excessive happiness disorder



Sample Size: Café Rule 1

The 50:50 Rule (proportions)

Control Rate Number Events

Control#(Rule 1)

Control#(Fisher exact)

20% 50 250 215

10% 50 500 463

5% 50 1000 962

50 events are needed in the control group:

(For an 80% chance of finding a 50% reduction)

Glasziou P, Doll H. Was the study big enough? Two cafe rules. Evid Based Med. 2006;11(3):69-70.




There is usually a 12% mortality rate

Y ou think your treatment will lower mortality

by 50%





There is usually a 12% mortality rate

Y ou think your treatment will lower mortality by 50%


12% means

12/100 or 24/200 or 48/400

and 50 per 417

Control + Treatment Groups = 834 in total



Systematic Review or meta-analysis?

A Systematic Review is a review of a clearlyformulated question that uses systematic andexplicit methods to identify, select and criticallyappraise relevant research, and to collect andanalyse data from the studies that are includedin the review.

Statistical methods (meta-analysis) may or maynot be used to analyze and summarize theresults of the included studies.



Is the review any good?

FAST appraisal

Question What is the PICO?

Finding Did they find most studies?

Appraisal Did they select good ones?

Synthesis

What to they all mean? Transferability of results



Why do I need to check the review?

Most reviews do not pass minimum criteria

A study of 158 reviews*

Only 2 met all 10 criteria

Median was only 1 of 10 criteria met

*McAlister Annals of Intern Med 1999

FAST tool = 4 criteria

APPRAISE SYNTHESISE TRANSFERABLEFIND



OUR MISSION: Are steroids

helpful for arthritis?

For osteoarthritis of the knee, are intra-articularinjections of corticosteroids helpful?

Find: what is your search strategy?



What it the review question (PICO)?

Population

Intervention

Comparison Outcome(s)




Do pedometers increase

activity and improve health?

Find: what is yoursearch strategy?

Databases?

Terms?

Other methods?

Do yourself then

Get neighbour¶s help



FIND: Did they find all Studies?

Check for existing systematic review?

Good initial search Terms (text and MeSH)

At least 2 Databases: MEDLINE, EMBASE, CINAHL, CCTR, ...

Plus a Secondary search Check references of relevant papers & reviews and

Find terms (words or MeSH terms) you didnt use

Search again! (snowballing)




Problems with searching

Finding overpublished work

Duplicate publications common

Finding unpublished work Negative trials unpublished?




Sources of studies

Near patient testing: 75 articles

0

10

20

30

40

50

SciCitation MEDLINE Experts HandSearch

Unique Non-unique




Is finding all published studies enough?

Negative studies less likely to bepublished than Positive

How does this happen?

Follow-up of 737 studies at Johns Hopkins(Dickersin, JAMA, 1992)

Positive SUBMITTED more than negative(2.5 times)




Registered vs Published StudiesOvarian Cancer chemotherapy: single v combined

Published Registered

No. studies 16 13Survival ratio 1.16 1.05

95% CI 1.06-1.27 0.98-1.12

P-Value 0.02 0.25

Simes, J. Clin Oncol, 86, p1529




Which are biased? Which OK?

1. All studies published in BMJ, Lancet, JAMA or NEJM

2. All publicly funded studies

3. All studies with more than 100 patients4. All studies conducted in the Northern Hemisphere

5. All studies registered studies




Publication Bias: Solution

All trials registered at inception,o The National Clinical Trials Registry: Cancer Trials

o National Institutes of Health Inventory of ClinicalTrials and Studies

o International Registry of Perinatal Trials

Meta-Registry of trial Registries

www.controlled-trials.com




APPRAISE & select studies

Did they select only thegood quality studies?




Miscalculating Number

Needed to Treat (NNT)

1. EFFICAC Y : Would the death penalty formiscalculating an NNT decrease miscalculation?

es/No2. ATTITUDE: Should we introduce the death

penalty for miscalculation of NNT? Yes/No




-2

-1

0

1

2

Deterrent Efficacy Attitude

A g a i n s t

I n

F a v o u r

Death Penalty: beliefs and contradictory studies

Proponents

Opponents

Selective Criticism of EvidenceBiased appraisal increases polarization

Lord et al, J Pers Soc Psy, 1979, p2098




Selective Criticism of Evidence

Positive Negative

Relevance 5.2 4.9

Methods 4.2 2.4

Presentation 4.3 2.6

Summary 3.2 1.8

28 reviewers assessed one ³study´

results randomly positive or negative

(Cog Ther Res, 1977, p161-75)




Assessment: How can you avoid biased

selection of studies?

Assessment and selection should be:

Standardized Objective OR

Blinded to Results

* assessment of quality blind to study outcome




R isk Factors

Cause(s)

Symptoms

Signs, Tests

Prognosis

Treatment Effect

Past current future

Types of question: stroke

Frequency

Cohort Study SurveyInception Cohort Study

Treatments

Randomised Trial



Was it a fair race?

1. Fair start?

2. Few drop outs?

3. Fair finish?



Participants

Intervention Group (IG) &Comparison Group (CG)

Outcome

IG CG

+ -+

- DC

BA

Representative?

Allocation?

Selection?

Maintenance of allocation?

VALIDITY

Maintenance?treated equally?compliant?

Measurementsblind subjective?

ORobjective?

QUESTION:

Allocation?Randomised?comparable groups?

Measurement of outcomes?

DESIGN:



YOUR MISSION: Are steroids

helpful for arthritis?

For osteoarthritis of the knee, are intra-articularinjections of corticosteroids helpful?

Assess: what inclusion criteria would youuse?




Synthesis: pooling the results





Meta-analysis (Forest) plot


Graphical and tabular summary of studies?




Transferable? Use in my patients

Is the AVERAGE effect similar across studies?

If NO, then WH Y ? Study methods (RAMbo - biases)

PICO (Patients, Intervention, )

If YE

S, then 2 questions Effect in different individuals?

Which version of treatment?





Meta-analysis (Forest) plot

Are the results similar across studies? 3 tests1. Eyeball test do they look they same?2. Test of Null hypothesis of no variation (p-value)3. Proportion of variation not due to chance (I2)





Are these trials different?




Epidemiology of Reviews



Conclusion

EBM and Systematic Review

EBM (quick & dirty)

Steps1. Ask Question

2. Search3. Appraise

4. Apply

Time: 90 seconds < 20 articles

This patient survives!

Systematic Review

Steps1. Ask Question

2. Search ++++ x 23. Appraise x 2

4. Synthesize

5. Apply

Time: 6 months, team < 2,000 articles

This patient is dead

Find a systematic review!! (and appraise it F AST)



Pros and cons of systematic

reviews

Advantages

Larger numbers & power

Robustness across PICOs

Disadvantages

May conclude small biases

are real effects



Is the review any good?

FAST appraisal

Question What is the PICO?

Finding Did they find most studies?

Appraisal Did they select good ones?

Synthesis What to they all mean?

Transferability of results



Combined results




Using review results:

what do I do with my patient?

STUD Y : meta-analysis of behaviouralinterventions for insomnia adults .. confirms the efficacy of behavioral

interventions for person with chronic insomnia.

PROBLEM: No regimens for behaviouralintervention described Author asked: what specific treatment regime

(or regimes) would you recommend based on

your review? Author response: It was found that cognitive,

behavioral and relaxation therapies all in generallead to similar improvements in sleep outcomes---although cognitive approaches might have beena bit better. The references for these studies are

found in the article.

Rx

³Behavioural

Intervention´




Summary: systematic reviews

Advantages

Larger numbers & power

Robustness across PICOs

Disadvantages

May conclude small biases

are real effects



The results:

Are studies similar?

What are the overall results?

Similarity of results

Heterogeneity statistic

Similarity of question (PICO)

Y our judgement!



Traditional cf systematic

reviews

Traditional

Many questions

No search methods No inclusion criteria

No combining studies

Systematic

One question

Explicit search

Explicit inclusion criteria

Combine study results(meta-analysis)

Which are



(a*+) (b+)

(c*) (d)

Minimum clinical

Important difference

No difference

Which are

(i) statistically significant * and

(ii) Clinically significant +?



Which studies (presented as Odds Ratio):

Are not statistically significant?

Have < 50 patients in the control arm?



Sample Size: Cafe Rule 2 - continuous

the 17/(SD squared) Rule

Differencein mmHg Difference(in SDs) Number perarm

16 1 17

8 ½ 68

4 ¼ 272

For continuous outcomes, number per arm is

17 / (³worthwhile difference´ measured in SDs)



In a small randomized double-blind trial of a new treatment for

acute myocardial infarction, the mortality in the treated group

was half that in the control group, but the difference was notsignificant. We can conclude that:

a) The treatment is useless

b) there is no point in continuing to develop the treatment c) the reduction in mortality is so great that we should

introduce the treatment immediately

d) we should keep adding cases to the trial until the Normaltest for comparison of two proportions is significant

e) we should carry out a new trial of much greater size



The best evidence for

different types of question

Level Treatment Prognosis Diagnosis

I SystematicReview of

SystematicReview of

SystematicReview of

II Randomisedtrial

InceptionCohort

Crosssectional

III



Has the systematic reviewer

done a good job?




Are the studies equivalent?

Are variations in results between studiesconsistent with chance?

(Test of homogeneity: has low power)

If NO, then WH Y ?

Variation in study methods (RAMbo - biases)

Variation in PICO (Patients, Intervention, )



Is bed rest ever helpful?A systematic review of trials*

10 trials of bed rest after spinalpuncture

no change in headache with bed rest Increase in back pain

Protocols in UK neurology units -

80% still recommend bed rest afterLP

Serpell M, BMJ 1998;316:170910

systematic reviews pg 2009

Documents