1

Review

Systematic Reviews and Meta-analysis

Trials

Steven Goodman, MD, PhD

Systematic Review

A review that has been prepared using a systematic approach to minimize biases and random errors.

- Use predefined and explicit methodology. - Review process: i) focused question, ii) search, iii) study selection, iv) quality appraisal, v) data extraction, vi) data analysis

Chalmers and Altman, 1994

Meta-analysis

A statistical technique for combining results of several studies into a single numerical estimate.

Also called statistical pooling.

Chalmers and Altman, 1994

Definitions cont’d

Therefore, you can have a systematic review without a meta-analysis, but you should NOT have a meta-analysis without a systematic review.

Why Perform a Systematic Review?  To start an investigation  To write a grant proposal  To write a manuscript  To write a thesis  To develop or choose a health policy  To talk on a topic (e.g. testify)  To write a paper summarizing the

review

Who is doing them?   Fellows   Faculty   EPCs   Govt Agencies   IOM   Professional societies   Guideline panels   Insurance Companies   Lawyers   Everyone (except many who should!)

2

SRs I have known   Agent Orange and cancer   Thimerosal and autism   ICDs and mortality   IVF and birth outcomes   Mammography and breast cancer   Treatment of PTSD in Veterans   Gene expression tests for breast

cancer prognosis

The Problem

Areas of concern   Clinical practice, or justification of RCT’s, does

not reflect evidence that is already conclusive.   Clinical practice has become entrenched and

clinical trials difficult, in spite of absent or poor evidence.

  Literature reviews based on a non-comprehensive, non-systematic searching and evaluation of the literature, followed by inappropriate mathematical methods.

The case of aprotinin

The aprotinin story (Fergusson et al., Clinical Trials, 2005)   Drug to reduce blood loss during surgery.   64 studies, 1987 and 2002, N=20 to 1784,

median 63.   Efficacy established almost immediately, effect

stabilized by 1992, after 12 studies.   Median of 20% of prior trials cited, only 7/44

(15%) trials cited the largest one.   Systematic reviews in 1992 and 1997 were cited

only 3 times.

3

1/3 1

Aprotinin worse

Aprotinin better

Citation pattern for 64 Aprotinin studies

“The scandalous failure of science to cumulate evidence scientifically” Iain Chalmers, Clinical Trials, 2:12-14, 2005.

“The article by Dean Fergusson and his colleagues ... is the most recent evidence of an ongoing scandal in which research funders, academia, researchers, research ethics committees and scientific journals are all complicit. New research should not be designed or implemented without first assessing systematically what is known from existing research. The failure to conduct that assessment represents a lack of scientific self-discipline that results in an inexcusable waste of public resources. In applied fields like health care, failure to prepare scientifically defensible reviews of relevant animal and human data results not only in wasted resources but also in unnecessary suffering and premature death.”

Albumin Meta-analysis

4

Albumin meta-analysis reaction Albumin meta-analysis reaction

BMJ Editorial “Every 100 patients treated means six more deaths. If true, this is deeply

disturbing.”

London Observer “300 die as health chiefs dither.”

“Burns cure kills thousands”

Some breast cancer examples

Cochrane Final Summary: 13 Yr Total Mortality

Cochrane Summary: 13 Yr Breast Ca Mortality

5

Cochrane Summary: 13 Yr All Cancer Mortality

Cochrane Summary: Mastectomy + Lumpectomy

M+L Rate in Screening arm: 1.63% M+L Rate in No Screening arm: 2.15%

Difference = 0.5%

Cochrane Summary: Mastectomies alone

Problems with non-systematic reviews   Noncomprehensive search and no

criteria for inclusion   Only English language   No methodological quality

assessment   Don’t properly account for

quantitative weight of evidence (a lot of “vote counting”

  Outdated   Often opinion pieces parading as

objective evidence summaries

Meta-analytic principles • Different studies addressing the

same question are estimating approximately the same effect.

• Each study contributes to the net weight of evidence - they are not "positive" or "negative"

How to interpret a plot

Treatment better Control better

Point estimates+CI (from individual trials)

Pooled estimate+CI (meta-analysis)

1

1

2

3

6

A simple example of meta-analysis

 Two studies looking at the 5 year mortality in subjects on Peritoneal Dialysis vs. Hemodialysis

A simple example of meta-analysis

Trial #1: N=160 PD 5 yr Mortality: 15% HD 5 yr Mortality: 25% Statistics: 10% difference, ±13%

p = 0.17, "Non-significant" Trial #2: N=300

PD 5 yr Mortality: 10% HD 5 yr Mortality: 15% Statistics: 5% difference, ±7%

p = 0.22, "Non-significant"

A simple example of meta-analysis

-5

0

5

10

15

20

25

Pooled Effect

PD better HD better

Steps in a meta-analysis • Assemble team of content and methods experts • Literature search • Select articles by defined criteria • Summarize qualitative design characteristics in "evidence

tables" • Assess and describe qualities of evidence • Decide whether studies can be pooled:

Qualitative and Quantitative criteria • Pool data if appropriate, w/ stat. model and measure of

effect • Conclusions, with appropriate caveats

Problems for all systematic reviews   Publication Bias

  Investigators are the main source   Refers to both delay and non-publication

  Qualitative heterogeneity   Designs or patients are different enough so

Total evidence < S Indiv. study evidence   Quantitative heterogeneity

  Effect sizes that differ beyond what chance might produce.

Heterogeneity

7

Heterogeneity Evidence Tables

The Cochrane Collaboration What it does and how it does it...

The Cochrane Collaboration aims to help people make well-informed decisions about healthcare by

preparing, maintaining and promoting the accessibility of systematic reviews of the effects of

healthcare interventions

Archie Cochrane

1993 - First Cochrane Colloquium. Oxford, England

8

It’s about collaboration

13,000 active contributors 90 countries

Working together toward a common goal

The Cochrane Library

  Published quarterly by John Wiley, Inc   Cochrane Database of Systematic

Reviews (4677 reviews and protocols)

  Other databases   CENTRAL Register of Controlled Trials (489,167)   Methodology Register (9048)   Database of Methodological Reviews (19)   Database of Abstracts of Reviews of Effectiveness (5931)   NHS Health Technology Assessments (6358)

  http://www.welch.jhu.edu/

9

So how is all this work organized?

www.cochrane.org

Steering Group

Centers Review Groups

Fields

Networks Methods Groups

Secretariat Cochrane Centers (12) and Branches (15)

  Australasian Centre   Thai Network, New Zealand Branch, Singapore Branch, South Asian Branch

  Brazilian Centre   Canadian Centre   Chinese Centre

  Hong Kong Branch   Dutch Centre

  Belgian Branch   German Centre   Iberoamerican Centre

  Colombian Branch, Venezuelan Branch   Italian Centre   Nordic Centre

  Finnish Branch, Norwegian Branch, Russian Branch   South African Centre

  Nigerian Branch   UK Centre

  Bahrain Branch   US Center

  Baltimore Center, Boston Branch, San Francisco Branch

Cochrane Review Groups (CRGs) (n=51)

Cochrane Acute Respiratory Infections Group Cochrane Airways Group Cochrane Anaesthesia Grop Cochrane Back Group Cochrane Bone, Joint and Muscle Trauma Group Cochrane Breast Cancer Group Cochrane Childhood Cancer Group Cochrane Colorectal Cancer Group Cochrane Consumers and Communication Group Cochrane Cystic Fibrosis and Genetic Disorders Group Cochrane Dementia and Cognitive Improvement Group Cochrane Depression, Anxiety and Neurosis Group

Cochrane Developmental, Psychosocial and Learning Problems Group Cochrane Drugs and Alcohol Group Cochrane Ear, Nose and Throat Disorders Group Cochrane Effective Practice and Organisation of Care Group Cochrane Epilepsy Group Cochrane Eyes and Vision Group Cochrane Fertility Regulation Group Cochrane Gynaecological Cancer Group……………

Cochrane Fields (n = 13) Behavioral Medicine Field Campbell and Cochrane Health Equity Field Cancer Network Child Health Field Complementary Medicine Field Health Care of Older People Field Health Promotion and Public Health Field Neurological Network Occupational Health Field Prehospital and Emergency Health Field Primary Health Care Field Rehabilitation and Related Therapies Field Vaccines Field

10

Methods groups (n = 11) Applicability and Recommendations Methods Group

Campbell and Cochrane Health Economics Methods Group

Individual Patient Data Meta-analysis Methods Group

Information Retrieval Methods Group

Non-Randomised Studies Methods Group

Patient Reported Outcomes Methods Group

Prospective Meta-Analysis Methods Group

Qualitative Methods Group

Bias Group

Screening and Diagnostic Tests Methods Group

Statistical Methods Group

Consumer Network

Base/Convenor in the US   USCC (Baltimore is main contact); also San Francisco,

Boston   Review Editorial bases and Satellites

  Prostatic Diseases and Urologic Cancers (Minneapolis)   HIV/AIDs/STDs (San Francisco)   Neonatal (Burlington, Vt)   Pain, Palliative and Supportive Care – Headache satellite

(Durham, NC)   Eyes and Vision (US satellite) – (Baltimore)

  Fields   Complementary Medicine (Baltimore)   Behavioral Medicine (New York)   Primary Care (New York)

  Methods Groups   Screening and Diagnostic Tests (Providence, RI)

  Consumers   [Consumers United for Evidence-Based Healthcare

(Baltimore)]

Getting involved in the US   Visit www.cochrane.org and www.cochrane.us   Make sure your institution subscribes to The Cochrane

Library (CLib)   Get into CLib, get to know how it works and how it can

help you and your work   Attend the Colloquium (October 2007 in Sao Paolo)   Subscribe to CCInfo   Sign up for a workshop (how to do a systematic review,

what is evidence-based healthcare, handsearching, etc)   Handsearch a journal   Volunteer to be a peer reviewer   Find the review group, field, methods group that best

meets your needs   Co-author a review   Get on the USCC’s contact list

11

2003 Barcelona – Iberoamerican Center staff serenade 2003 Barcelona – Gaudi welcomes Cochrane

Info and Contact

www.cochrane.org www.cochrane.us uscc@jhsph.edu