Meta-Analysis for Clinical Researchers

An Introduction to Systematic Reviews

& Meta-analysis

Topics

• Methods for pooling evidence across independent studies Pooling binary and continuous outcomes

• Differences between fixed and random effects models Guidelines for appraising published systematic reviews/meta-analyses

Why Meta-analysis/Systematic Reviews?

• “. . . the mass of new information makes it difficult for practicing physicians to follow the literature in all areas that might be relevant to their practices. New methods to synthesize and present information from widely dispersed publications are needed . . . .”

Jerome Kassirer. Clinical trials and meta-analysis: what do they do for us? N Engl J Med 1992; 327:273-4.

Why Need Meta-analysis? Information Explosion

• 10-fold Increase in Number of Professional Journals

• Psychology Journals: 91 (1951) --> 1,175 (1992)

• Math Science Journals: 91 (1953) --> 920 (1992)

• Biomedical Journals: 2,300 (1940)--> 23,000 (1993)

Problem – Conflicting Information

• Not only is there more information, but . . .

• Not all information is of equal quality

• Information does not necessarily = evidence

• There is often conflicting information & reports Traditional narrative reviews can be very “impressionistic”

Problems With Traditional Literature Reviews Addressed in Meta-analysis

• Selective inclusion of studies, often based on the reviewer's own impressionistic view of the quality of the study

• Differential subjective weighting of studies in the interpretation of a set of findings

• Misleading interpretations of study findings • Failure to examine characteristics of the studies

as potential explanations for disparate or inconsistent results across studies

• Failure to examine moderating variables in the relationship under examination

Brief History of Meta-analysis (very abbreviated)

• 1900’s: Karl Pearson pooled correlation coefficients -enteric fever & inoculation rates British Army

• 1930’s: Ronald Fisher pooled p-values • 1972: Richard Peto: log rank test for combining (binary)

data from different trials; Mantel, Thomas Chalmers • 1976: Gene Glass “meta-analysis” effects of

psychotherapy • 1989 Murray Enkin, Marc Keirse, Iain Chalmers Effective

Care in Pregnancy & Childbirth • 1992/1993 UK Cochrane Centre/Cochrane Collaboration

created

Primary, Secondary and Meta-analysis of Research

• “Primary analysis is the original analysis of data in a research study . . . . Secondary analysis is the re-analysis of data for the purpose of answering the original research question with better statistical techniques, or answering new questions with old data . . . .” GV Glass 1976, p. 3

Primary, Secondary and Meta-analysis of Research

• “Meta-analysis refers to the analysis of analyses . . . . the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature.” GV Glass 1976, p. 3

Rationale for Systematic Reviews

• “provide summaries of what we know, and do not know, that are as free from bias as possible.” (Chalmers et al 1999)

• “research that uses explicit & transparent methods to synthesise relevant studies, allowing others to comment on, criticise or attempt to replicate the conclusions reached. Systematic reviews follow same set of procedures as any individual study, & are often reported in the same way. . . .” (Petrsino et al 1999)

4 Basic Questions That a SR/MA Tries to Answer

• Are the results of the different studies similar? • To the extent that they are similar, what is the

best overall estimate of effect? • How precise and robust is this estimate? • Can dissimilarities be explained? • Lau J, Ioannidis JPA, Schmid CH. Quantitative

Synthesis in Systematic Reviews. Annals of Internal Medicine 1997; 127:820-826.

What is a Systematic Review?

• State objectives of the review, & outline eligibility criteria Search for studies that seem to meet eligibility criteria Tabulate characteristics of each study identified & assess its methodological quality Apply eligibility criteria & justify any exclusions

What is a Systematic Review?

• Assemble the most complete dataset feasible, with involvement of investigators

• Analyse results of eligible studies. Use statistical synthesis of data (meta-analysis) if appropriate & possible

• Perform sensitivity analyses, if appropriate & possible (including subgroup analyses)

• Prepare a structured report of the review, stating aims, describing materials & methods, & reporting results

Cochrane Collaboration

Comparison with human genome project in potential impact for clinical medicine

• Naylor CD. Grey zones of clinical practice: some limits to evidence-based medicine. Lancet 1995; 345:840-2.

• What is the Cochrane Collaboration & why is it important?

• Cochrane Library CD (& WWW)

• Cochrane Database of Systematic Reviews (CDSR)

• Database of Abstracts of Reviews of Effectiveness (DARE)

• Cochrane Central Register of Controlled Trials (CENTRAL)

• Cochrane Review Methodology Database

• Health Technology Assessment DB (HTA)

• NHS Economic/Evaluation Database (NHS EED)

Cochrane Library CD (& WWW)

Sources of Evidence in “Real-time”: Cochrane Library (Issue 1, 2003)

• CDSR (Cochrane Database of Systematic Reviews) – >50 CRGs, >1500 complete reviews, >1200

protocols, >200 new reviews/year • DARE (DB of Abstracts of Reviews of

Effectiveness) >3800 abstracts • Cochrane Central Register of Controlled Trials

(CENTRAL) >353,000 RCTs/CCTs • HTA (Health Technology Assessment Database)

>2800 • NHS Economic /Evaluation DB >10,000

Retrieval Problems

– 50% of RCTs not found in MEDLINE – 44% of all RCTs (17% - 76%)

• 72% of RCTs in MEDLINE (32% - 91%) – Publication Type (PT) indexing – RCT (1991) – CCT (1995) – Meta-analysis (1993)

Hand Searching Medical Journals

• >630 journals

• 19,266 RCTs tagged (1991-93)

• 14,964 CCTs tagged (1985-90)

• MEDLINE updated annually to include or re-tag these studies as RCTs or CCTs

Distinguishing Feature of Cochrane Reviews

• “The fundamental distinction between Cochrane Reviews and other reports of systematic reviews is that their authors are expected to update and amend them in the light of relevant additional data, and criticisms or other comments.”

Sir Iain Chalmers 1999

Effects of Asthma Self Management Educational Interventions in Children and Adolescents: A Systematic Review and Meta-analysisJames P Guevara, MD,MPHa, Fredric M Wolf, PhDb, Cyril M Grum,

MDc, & Noreen M Clark, PhDc aUniversity of Pennsylvania, Philadelphia, USA

bUniversity of Washington, Seattle, USA cUniversity of Michigan, Ann Arbor, USA

Cochrane Library 2003 (1); BMJ 2003; in press.

Objectives/Purpose

• To determine the effectiveness of asthma self-management education programs on health outcomes in children

• Outcomes: • Lung Function • Asthma Morbidity • Heath Care Utilization: • Self-reported Perceptions of Self-care Abilities

Objectives/Purpose

– To conduct subgroup analyses to examine the impact of

– type of educational intervention (individual vs. group) – intensity of educational intervention (no. of sessions) – self-management strategy (symptom vs. peak flow) – degree of asthma severity – length of follow-up – study quality (adequacy of allocation concealment,

withdrawal rates, and type of trial, i.e., RCT vs. CCT)

Inclusion Criteria for Studies

• Randomized controlled trials (RCTs) or controlled clinical trials (CCT)

• Children & adolescents ages 2 to 18 years old • Educational intervention designed to teach one

or more self-management strategies related to prevention, attack management, or social skills

• Included outcomes on pulmonary function tests, morbidity, or health care utilization

Search Strategy – References & Databases

• Studies were identified from – Cochrane Airways Group's Special Register of

Controlled Trials comprised of references from – MEDLINE (1966-2000) – EMBASE (1980-2000) – CINAHL (1982-2000)

• hand searched airways-related journals • PsychINFO • Reference lists from relevant review articles that

were identified (ancestry approach)

Search Strategy - Terms

• asthma OR wheez* AND

• education* OR self management OR self-management AND

• placebo* OR trial* OR random* OR double-blind OR double blind OR single-blind OR single blind OR controlled study OR comparative study.

Identification of Trials

• Potentially relevant studies from literature search and hand searches (n=318)

• Excluded on basis of abstract, e.g., not randomised or controlled clinical trials (n=273) Articles selected for full text review (n=45)

• Excluded after full text review (n=13)

• Eligible trials (n=32)

Main Outcome Measures

– Lung Function (pulmonary function tests): – FEV1

• PEFR (Peak Flow ) – Asthma Morbidity: – asthma exacerbations – days of school absence – days of restricted activity – nights disturbed by asthma – asthma severity

Main Outcome Measures

– Health Care Utilization: – physician visits – emergency department (ED) visits

• hospitalizations – Self-reported Perceptions of Self-care

Abilities: – self-efficacy

Subgroup Analyses

• Time of enrollment in the intervention (1-6 mo, 7-12 mo, or >/=12 months) Self-management strategy (peak flow-based vs. symptom-based) Intervention type (individual vs. group) Intensity of intervention (single vs. multiple sessions) Study quality (RCT vs. CCT; random allocation procedures)

Sources of Bias

• RCTs (primary studies) – Selection bias – Performance bias – Exclusion bias – Detection bias

• Meta-analyses – Publication bias – Language bias – Coder bias “Apples & oranges” vs “fruit”

(heterogeneity) – Quality bias (small studies) – Multiple publication bias

32

An Inverted Funnel Plot to Detect Publication Bias

33

An Inverted Funnel Plot to Detect Publication Bias

Pooling Binary & Continuous Outcomes

– General principals – Effect size – Confidence Intervals – Types of data – Sources of potential bias

• Interpreting results • Examples

Differences Between Fixed & Random Effects Models

• So what’s in a model?

• Why does it matter?

• How to deal with heterogeneity?

Heterogeneity

• Common, to be expected, not the exception • Should do test for homogeneity, but . . . interpret

heterogeneity cautiously in spirit of exploratory data analysis – Exploring sources of heterogeneity can lead to

insights about modification of apparent associations by various aspects of

– Study design – Exposure measurements

– Study populations

Heterogeneity

• Relations discovered in process of exploring heterogeneity may be useful in planning & carrying out new studies

• Excluding outliers solely on basis of disagreement with other studies can lead to seriously biased summary estimates (avoid)

• Easier to interpret sources of heterogeneity when identified in advance of data analysis (not when suggested only by data)

Fixed & Random Effects

• Fixed effects models assume that an intervention has a single true effect

• Random effects models assume that an effect may vary across studies

Random Effects

• Assumes sample of studies randomly drawn from population of studies

• This is NOT typically true because: – All trials are included

– Trials are systematically (e.g., conveniently) sampled and not randomly sampled

Random Effects

• Primary value of M-A is in search for predictors of between-study heterogeneity

• Random-effects summary is last resort only when predictors or causes of between-study heterogeneity cannot be identified

• Random-effects can conceal fact that summary estimate or fitted model is poor summary of the data Sander Greenland.

Am J Epidemiol 1994;140;290-6.

Random Effects

• Sometimes needed, but more sensitive to publication bias than fixed-effects

• Random effects weights vary less across studies than fixed-effects weights

• W = 1/v versus w = 1/(v + t2) • Leads to reduced variation in weights • Thus smaller studies given larger relative weights when

random effects models used • Thus influenced more strongly by any tendency NOT to

publish small statistically insignificant studies biased estimate, spuriously strong associations

Random Effects

• Fixed effects weights vs. random effects weights • W = 1/v versus w = 1/(v + t2) • Identical when there is little or no between study

variation • When differ, confidence intervals are larger for

random-effects than fixed effects • Smaller studies given larger relative weights in

random effects models & > influence • Conversely, influence of larger studies is less • May result in type II (beta error), e.g., Finding no

significant difference when one truly exists

43

Nights Disturbed by Asthma

3 trials, n = 202 (SMD –0.34, 95% CI –0.62 to –0.05) fixed effects (SMD

–0.39, 95% CI –1.07 to +0.28) random effects

44

Methodologic Choices & Their Implications in Dealing With Heterogeneous Data

in a Meta-analysis

Lau J, Ioannidis JPA, Schmid CH. Quantitative Synthesis in Systematic Reviews. Annals of Internal Medicine 1997; 127:820-826.