META-ANALYSIS: THE ART AND SCIENCE OF COMBINING INFORMATION

Ora Paltiel, October 28, 2014

DEFINITIONS

• The statistical analysis of a large collection of results from individual studies for the purpose of integrating the findings

• A quantitative review and synthesis of results of related but independent studies

• “overview”• “data pooling”• “data synthesis”• systematic review

“Meta “

• Webster’s dictionary:

a) occurring later than or in succession to

b) situated behind or beyond

c) change, transformation

Examples: metaphysics ,

metamorphosis .

OVER 2 MILLION MEDICAL ARTICLES ARE PUBLISHED EACH YEAR.

The Problem

The findings of new studies not only “ differ from previously established truths but disagree with one another, often violently”

-Morton Hunt, How Science Takes Stock, P.1

The Goal of Meta-Analysis:“Making Order of Scientific Chaos”

• Began as a tool in Social Sciences21 citations in 1986431 citations in 1991

more than 45000 todayIn Medicine – at first only RCTsNow – thousands of meta-analyses of

observational studies

• is a group of over 15,000 volunteers in more than 90 countries who review the effects of health care interventions tested in biomedical randomized controlled trials

• reviews have also studied the results of non-randomized observational studies.

• The results of these systematic reviews published as "Cochrane Reviews" in the Cochrane Library

• Founded in 1993 under the leadership of Iain Chalmers. • developed in response to Archie Cochrane's call for up-to-date,

systematic reviews of all relevant randomized controlled trials of health care.

Cochrane collaborationGoal : to help people make well informed decisions about health care by preparing, maintaining and ensuring the accessibility of

systematic reviews of the effects of health care interventions. The principles of the Cochrane Collaboration are:

• collaboration • building on the enthusiasm of individuals • avoiding duplication • minimizing bias • keeping up to date • striving for relevance • promoting access • ensuring quality • continuity • enabling wide participation

Major goals of Meta-Analysis

• Objective summaries• Increase power to detect true effects• Estimate effect size • Resolve uncertainty• Explore heterogeneity and reasons for it

If the studies produced dissimilar results, How did they differ? Why? Study design, quality, populations, subtle intervention differences etc

• Tool for conducting evidence-based medicine and for setting policy

How to do a Meta-Analysis

1. Define research question, including intervention, population, and outcome to be assessed

2. Define eligibility criteria (types of study, design)3. Identify all studies (published or un) which deal with

the specified problem4. Evaluate each article for inclusion or exclusion, on the

basis of predefined criteria5. Summarize, numerically, the results of these studies6. Interpret these findings, with emphasis on explaining

differences as well as summarizing the data

Literature review• A comprehensive, systematic literature review

should be conducted• Sources: citation indexes, abstract databases,

clinical trials registers, references , • Issues: language, “grey literature”, conference

abstracts, unpublished findings• Meta-analysis is research, which should be

reproducible, methods incl key words must be able to be replicated

Problem of publication biaspublication bias

SEARCH STRATEGY- example

Horvath et al BMJ 2010;340:c1395

Information Assembled

• The report ( author, year)• The study (population)• The patients (demographic and clinical

characteristics)• The design • The treatment • The effect size ( estimate , SE)Methods, reliability and validity of recording

information need to be documented

• “Head-Counting - Statistical”: Count the number of significant results in each direction Result: 6 favor treatment, 0 favor placebo, 27 nonsignificant

• “Head-Counting”: Count the direction of the results in the studiesResult: 24 favor treatment, 9 favor placebo

Thirty three trials of streptokinase vs. conventional treatment for Acute Myocardial Infarction

Streptokinase - Summary

• Streptokinase reduces mortality by about 22%

• Efficacy proven by 2 large RCTs in 1986 and 1988

• Meta-analysis proved efficacy in 1971

• 6380 lives could have been saved in large RCTs alone

What can we learn from the Forest Plot?

Meta-analysis of gestational diabetes outcomes – 1. Maternal

Horvath et al BMJ 2010;340:c1395

Statistical Methods

• We have a series of measures of association, one for each study

• We wish to summarize these measures• This can be carried out using a weighted

average of the estimates taken from each study.

Classic Meta-Analysis

• Analyzes RR, OR, or absolute differences in percentages between groups.

• Uses the the inverse of the variance of the

estimate provided by each participating trial for the weights. This gives a minimum variance unbiased estimate of the effect.

• Large trials carry more weight than small trials.

Inference: fixed .v. random effects

If interest is centered on making inferences for the populations that have been sampled, and we assume that there is a single effect of treatment - then a fixed effects approach is

used .In this approach the only source of uncertainty is that resulting from sampling patients into the studies. Variation stems

from within-study variation study.

The population to which we wish to generalized the results consists of a set of studies having identical characteristics

Random-effects

• In random-effects approach the existing studies are considered as a random sample from a population of studies

• Random-effects approach is used when inferences are to be generalized to a population in which studies may differ in their effect and characteristics

• Random effects approach integrate also the between-study variability

Fixed vs. Random-effects

• The use of random-effects will produce somewhat larger 95% CI

• A good practice is to first perform a test of heterogeneity between studies. If no significant variation is found between studies - a fixed-effects approach can be used

• There are a number of ways to model random-effects

Heterogeneity

Horvath et al BMJ 2010;340:c1395

Sensitivity analysis- comparators or control groups

Sensitivity analysesexcluding studies with predefined less desirable

characteristics, as follows:

Risk of bias

When the analysis was limited to two studies with a low risk of bias for

random sequence generation and/or allocation concealment the add-on effect

of acupuncture on patient-reported global assessment remained significant (RR

0.39, 95% CI 0.18–0.88, I2 = 0%).

Sample size

When four studies with ≥ 40 participants per group were pooled, there was no

significant difference in the risk of symptoms persisting or worsening between

the acupuncture and control groups (RR 0.50, 95% CI 0.24–1.05, I2 = 55%).

Assessing Quality

A systematic approach should be used in order to assess the quality of the studies and to determine inclusion/exclusion of studiesExplicit methods limit bias in identifying and rejecting studies

Scales such as Jaddad scale

Domains to be assessed

• Methodological quality ( bias)

• Precision in estimation

• External validity

Assessing quality of included studies: -- RCTs- account in text

Assessment of bias, graphic representation

Risk of bias: Tabular presentation

Further Exploring Heterogeneity

• In case of substantial heterogeneity between studies, exploring its causes can be performed by considering covariates on the study level that could ‘explain’ differences between studies.

• Such analyses are called meta-regression

Meta-regression by study properties

Publication bias. Some studies are not published, selective presentation in those

published.Do a comprehensive search. Use a funnel plot

Publication bias use of the funnel plot

1-S

AM

PLE

SiZ

E

SAMPLE SiZE

Conclusions

• In times of increasing amount of information-a systematic approach to synthesizing information has many advantages.

• A systematic approach enables exploring heterogeneity between studies

• As any other type of research systematic review should be carried out methodically and cautiously

Problems with Meta-Analysis in Real Life

• “Meta-analysis” often not done, or very few studies combined

• Retrospective study• Publication Bias• Heterogeneity

Future

• Expect to see lots of meta-analyses

• Good ones and bad ones

• Scientific community will decide whether it is useful

Be skeptical of everything

Supplementary material

NEWCAS TLE - O TTAW A Q UALITY ASS ESS MENT SCALECO HORT S TUDIES

Note: A study can be awarded a ma ximum of one star for each numbered item within the Selection andOutcome categories. A maximum of two stars can be given for Comparability

Selection1) Representativeness of the exposed cohort

a) truly representative of the average _______________ (describe) in the community b ) somewhat representative of the average ______________ in the community c) selected group of users eg nurses, volunteersd) no description of the derivation of the cohort

2) Selection of the non exposed cohorta) drawn from the same community as the exposed cohort b) drawn from a different sourcec) no description of the derivation of the non exposed cohort

3) Ascertainment of exposurea) secure record (eg surgical records) b) structured interview c) written self reportd) no description

4) Demonstration that outcome of interest was not present at start of studya) yes b) no

Compara bility1) Comparability of cohorts on the basis of the design or analysis

a) study controls for _____________ (select the most important factor) b) study controls for any additional factor (This criteria could be modified to indicate specific

control for a second important factor.)

Outcome1) Assessment of outcome

a) independent blind assessment b ) record linkage c) self reportd) no description

2) Was follow-up long enough for outcomes to occura) yes (select an adequate follow up period for outcome of interest) b) no

3) Adequacy of follow up of cohortsa) complete follow up - all subjects accounted for

b ) sub jects lost to follow up unlikely to introduce bias - small number lost - > ____ % (select an adequate %) follow up, or description provided of those lost)

c) follow up rate < ____% (select an adequate %) and no description of those lostd) no statement

N EWCAS TLE - O TTAW A Q UALITY ASS ESS MENT SCA LECAS E CON TRO L S TUD IES

Note: A study can be awarded a ma ximum of one star for each numbered item within the Selection andExposure categories. A maximum of two stars can be given for Comparability.

Selection

1) Is the case definition adequate?a) yes, with independent validation b) yes, eg record linkage or based on self reportsc) no description

2) Representativeness of the casesa) consecutive or obviously representative series of cases b) potential for selection biases or not stated

3) Selection of Controlsa) community controls b) hospital controlsc) no description

4) Definition of Controlsa) no history of disease (endpoint) b) no description of source

Compara bility

1) Comparability of cases and controls on the basis of the design or analysisa) study controls for _______________ (Select the most important factor.) b) study controls for any additional factor (This criteria could be modified to indicate specific

control for a second important factor.)

Exposure

1) Ascertainment of exposure

a) secure record (eg surgical records) b) structured interview where blind to case/control sta tus c) interview not blinded to case/control status

d) written self report or medical record only

e) no description

2) Same method of ascertainment for cases and controlsa) yes b) no

3) Non-Response ratea) same rate for both groups b) non respondents describedc) rate different and no designation

Fixed versus Random effects

Robustness of results-meta-regressionMeta-regressions are similar in essence to simple regressions, in which an outcome variable is predicted according to the values of one or more explanatory variables. In meta-regression, the outcome variable is the effect estimate (for example, a mean difference, a risk difference, a log odds ratio or a log risk ratio). The explanatory variables are characteristics of studies that might influence the size of intervention effect

an investigation of how a categorical study characteristic is associated with the intervention effects in the meta-analysis. For example, studies in which allocation sequence concealment was adequate may yield different results from those in which it was inadequate. Here, allocation sequence concealment, adequate /inadequate, is a categorical characteristic at the study level. MR in principle allows the effects of multiple factors to be investigated simultaneously (although this is rarely possible due to inadequate numbers of

studies) (Thompson 2002). Meta-regression should generally not be considered when there are fewer than ten studies in a meta-analysis.