Basics of Meta-analysis

Steff Lewis, Rob Scholten

Cochrane Statistical Methods Group

(Thanks to the many people who have worked on earlier versions of this

presentation)

Introduction

3

Session plan• Introduction

• Effect measures – what they mean

– Exercise 1

• Meta-analysis

– Exercise 2

• Heterogeneity

– Exercise 3

• Summary

4

Before we start…this workshop will be discuss binary outcomes only…

• e.g. dead or alive, pain free or in pain, smoking or not smoking

• each participant is in one of two possible, mutually exclusive, states

There are other workshops for continuous data, etc

5

Where to start

1. You need a pre-defined question

• “Does aspirin increase the chance of survival to 6 months after an acute stroke?”

• “Does inhaling steam decrease the chance of a sinus infection in people who have a cold?”

6

Where to start

2. Collect data from all the trials and enter into Revman

For each trial you need:

The total number of patients in each treatment group.

The number of patients who had the relevant outcome in each treatment group

Effect measures – what they mean

8

Which effect measure?

• In Revman you can choose:

– Relative Risk (RR) = Risk Ratio,

– Odds Ratio (OR)

– Risk Difference (RD) = Absolute Risk Reduction (ARR),

9

Risk

• 24 people skiing down a slope, and 6 fall• risk of a fall

= 6 falls/24 who could have fallen

= 6/24 = ¼ = 0.25 = 25%

risk = number of events of interest

total number of observations

10

Odds

• 24 people skiing down a slope, and 6 fall• odds of a fall

= 6 falls/18 did not fall

= 6/18 = 1/3 = 0.33 (not usually as %)

odds = number of events of interest

number without the event

11

Expressing it in words

• Risk– the chances of falling were one in four, or

25%

• Odds– the chances of falling were one third of the

chances of not falling– one person fell for every three that didn’t fall– the chances of falling were 3 to 1 against

12

Do risks and odds differ much?

• Control arm of trial by Blum– 130 people still dyspeptic out of 164

• chance of still being dyspeptic– risk = 130/164 = 0.79; odds =130/34 = 3.82– Tanzania trial, control arm– 4 cases in 63 women

• chance of pregnancy induced hypertension– risk = 4/63 = 0.063; odds = 4/59 = 0.068

eg1 - Moayeddi et al BMJ 2000;321:659-64eg2 - Knight M et al. Antiplatelet agents for preventing and treating pre-eclampsia (Cochrane

Review). In: The Cochrane Library, Issue 3, 2000. Oxford: Update Software.

13

Comparing groups – 2x2 tables

Blum et al Still dyspeptic

Not still dyspeptic

Total

Treatment 119 45 164

Control 130 34 164

Total 249 79 328

14

Risk ratio (relative risk)

• risk of event on treatment= 119/164

• risk of event on control= 130/164

• risk ratio = 119/164 = 0.726 = 0.92130/164 0.793

= risk on treatmentrisk on control

Where risk ratio = 1, this implies no difference in effect

Blum

et al

Still dyspeptic

Not still dyspeptic

Total

Treat 119 45 164

Control 130 34 164

Total 249 79 328

15

Odds ratio

• odds of event on treatment= 119/45

• odds of event on control= 130/34

• odds ratio = 119/45 = 2.64 = 0.69130/34 3.82

= odds on treatmentodds on control

Where odds ratio = 1, this implies no difference in effect

Blum

et al

Still dyspeptic

Not still dyspeptic

Total

Treat 119 45 164

Control 130 34 164

Total 249 79 328

16

What is the difference between Peto OR and OR?

• The Peto Odds Ratio is an approximation to the Odds Ratio that

works particularly well with rare events

17

Expressing risk ratios and odds ratios

• Risk ratio 0.92– the risk of still being dyspeptic on treatment was

about 92% of the risk on control– treatment reduced the risk by about 8%– treatment reduced the risk to 92% of what it was

• Odds ratio 0.69– treatment reduced the odds by about 30%– the odds of still being dyspeptic in treated patients

were about two-thirds of what they were in controls

18

(Absolute) Risk difference

• risk on treatment – risk on control• for Blum et al

119/164 – 130/164 = 0.726 – 0.793 = -0.067usually expressed as a %, -6.7%

• treatment reduced the risk of being dyspeptic by about 7 percentage points

• Where risk difference = 0, this implies no difference in effect

19

What do we want from our summary statistic?

• Communication of effect– Users must be able to use the result

• Consistency of effect– It would be ideal to have one number to

apply in all situations• Mathematical properties

20

SummaryOR RR RD

Communication - + ++

Consistency + + _

Mathematics ++ _ _

Further info in “Dealing with dichotomous data” workshop.

Exercise 1

Meta-analysis

23

What is meta-analysis?

• A way to calculate an average• Estimates an ‘average’ or ‘common’ effect• Improves the precision of an estimate by

using all available data

24

What is a meta-analysis?

Optional part of a systematic review

Systematic reviews

Meta-analyses

25

When can we do a meta-analysis?

• When more than one study has estimated an effect

• When there are no differences in the study characteristics that are likely to substantially affect outcome

• When the outcome has been measured in similar ways

• When the data are available (take care with interpretation when only some data are available)

26

Averaging studies

• Starting with the summary statistic for each study, how should we combine these?

• A simple average gives each study equal weight

• This seems intuitively wrong• Some studies are more likely to give an

answer closer to the ‘true’ effect than others

27

Weighting studies

• More weight to the studies which give us more information– More participants– More events– Lower variance

• Weight is closely related to the width of the study confidence interval: wider confidence interval = less weight

28

For example

Deaths on hypothermia

Deaths on control

Weight (%)

Clifton 1992 1/5 1/5 3.6

Clifton 1993 8/23 8/22 21.5

Hirayama 1994 4/12 5/10 11.3

Jiang 1996 6/23 14/24 23.4

Marion 1997 9/39 10/42 30.0

Meissner 1998 3/12 3/13 9.7

29

Displaying results graphically

• Revman produces forest plots

30

there’s a label to tellyou what the comparisonis and what the outcomeof interest is

31

At the bottom there’sa horizontal line. This is the scale measuringthe treatment effect.Here the outcome is deathand towards the left thescale is less than one,meaning the treatmenthas made death lesslikely.

Take care to read whatthe labels say – things tothe left do not always meanthe treatment is better thanthe control.

32

The vertical line in themiddle is where thetreatment and control have the same effect – there is no differencebetween the two

33

For each study there is an id

The data foreach trial are here, divided into the experimental and control groups

This is the % weightgiven to thisstudy in the pooled analysis

34

•Each study is given a blob, placed where the data measure the effect.•The size of the blob is proportional to the % weight •The horizontal line is called a confidence interval and is a measure of how we think the result of this study might vary with the play of chance. •The wider the horizontal line is, the less confident we are of the observed effect.

The label above the graph tells you what statistic has been used

The data shown in the graph are also given numerically

35

The pooled analysis is given a diamond shapewhere the widest bit in the middle is located at the calculated best guess (point estimate), and the horizontal width is the confidence interval

Definition of a 95% confidence interval: If a trial was repeated 100 times, then 95 out of those 100 times, the best guess (point estimate) would lie within this interval.

36

Could we just add the data from all the trials together?

• One approach to combining trials would be to add all the treatment groups together, add all the control groups together, and compare the totals

• This is wrong for several reasons, and it can give the wrong answer

37

If we just add up the columns we get34.3% vs 32.5% , a RR of 1.06, a higher death rate in the steroids group

From a meta-analysis, we getRR=0.96 , a lower death ratein the steroids group

38

Problems with simple addition of studies

• breaks the power of randomisation• imbalances within trials introduce bias

39

*#

In effect we are comparingthis experimental group directlywith this control group – this is not a randomised comparison

40

*

The Pitts trial contributes 17% (201/1194) of all the data to theexperimental column, but 8% (74/925) to the control column.Therefore it contributes more information to the average death rate inthe experimental column than it does to the control column.There is a high death rate in this trial, so the death rate for the expt column is higher than the control column.

41

Interpretation - “Evidence of absence” vs “Absence of evidence”

• If the confidence interval crosses the line of no effect, this does not mean that there is no difference between the treatments

• It means we have found no statistically significant difference in the effects of the two interventions

42

Review: SteffComparison: 01 Absence of evidence and Evidence of absence Outcome: 01 Increasing the amount of data...

Study Treatment Control OR (fixed) OR (fixed)or sub-category n/N n/N 95% CI 95% CI

1 study 10/100 15/100 0.63 [0.27, 1.48] 2 studies 20/200 30/200 0.63 [0.34, 1.15] 3 studies 30/300 45/300 0.63 [0.38, 1.03] 4 studies 40/400 60/400 0.63 [0.41, 0.96] 5 studies 50/500 75/500 0.63 [0.43, 0.92]

0.1 0.2 0.5 1 2 5 10

Favours treatment Favours control

In the example below, as more data is included, the overall odds ratio remains the same but the confidence interval decreases.

It is not true that there is ‘no difference’ shown in the first rows of the plot – there just isn’t enough data to show a statistically significant result.

43

Interpretation - Weighing up benefit and harm

• When interpreting results, don’t just emphasise the positive results.

• A treatment might cure acne instantly, but kill one person in 10,000 (very important as acne is not life

threatening).

44

Interpretation - Quality• Rubbish studies = unbelievable results

• If all the trials in a meta-analysis were of very low quality, then you should be less certain of your conclusions.

• Instead of “Treatment X cures depression”, try “There is some evidence that Treatment X cures depression, but the data should be interpreted

with caution.”

Exercise 2

Heterogeneity

47

What is heterogeneity?

• Heterogeneity is variation between the studies’ results

48

Causes of heterogeneity

Differences between studies with respect to:• Patients: diagnosis, in- and exclusion criteria,

etc.• Interventions: type, dose, duration, etc.

• Outcomes: type, scale, cut-off points, duration of follow-up, etc.

• Quality and methodology: randomised or not, allocation concealment, blinding, etc.

How to deal with heterogeneity

1.Do not pool at all

2. Ignore heterogeneity: use fixed effect model

3.Allow for heterogeneity: use random effects

model

4. Explore heterogeneity: (“Dealing with

heterogeneity” workshop )

50

How to assess heterogeneity from a Revman forest plot

51

Statistical measures of heterogeneity

• The Chi2 test measures the amount of variation in a set of trials, and tells us if it is more than would be expected by chance

• Small p values suggest that heterogeneity is present

• This test is not very good at detecting heterogeneity. Often a cut-off of p<0.10 is used, but lack of statistical significance does not mean there is no heterogeneity

52

Statistical measures of heterogeneity (2)

• A new statistic, I2 is available in RevMan 4.2• I2 is the proportion of variation that is due to

heterogeneity rather than chance• Large values of I2 suggest heterogeneity• Roughly, I2 values of 25%, 50%, and 75%

could be interpreted as indicating low, moderate, and high heterogeneity

• For more info see: Higgins JPT et al. Measuring inconsistency in meta-analyses. BMJ 2003;327:557-60.

Fixed effect

Philosophy behind fixed effect model:• there is one real value for the treatment effect• all trials estimate this one value

Problems with ignoring heterogeneity:• confidence intervals too narrow

54

Random effects

Philosophy behind random effects model:• there are many possible real values for the

treatment effect (depending on dose, duration, etc etc).

• each trial estimates its own real value

 

Interpretation of fixed and random effects resultsIf there is heterogeneity, Fixed effect and Random effects models:• may give different pooled estimates • have different interpretations:

RD = 0.3: Fixed Effects ModelThe best estimate of the one and only real RD is 0.3

RD = 0.3: Random Effects ModelThe best estimate of the mean of all possible real values of the RD is 0.3

• Random Effects Model gives wider confidence interval• In practice, people tend to interpret fixed and random effects the

same way.

Exercise 3

Summary

58

Summary• Precisely define the question you want to answer

• Choose an appropriate effect measure

• Collect data from trials and do a meta-analysis if appropriate

• Interpret the results carefully– Evidence of absence vs absence of evidence

– Benefit and harm

– Quality

– Heterogeneity

59

Other sources of help and advice• The Reviewer’s handbook

– http://www.cochrane.org/resources/handbook/index.htm

• The distance learning material– http://www.cochrane-net.org/openlearning/

• The Revman user guide.– http://www.cc-ims.net/RevMan/documentation.htm

• The Collaborative Review Group you are working with