SYSTEMATIC REVIEWS AND META-ANALYSIS

Objectives

Define systematic review and meta-analysis

Know how to access appraise interpret

the results of a systematic review

Reasons to use reviews

Sheer volume of literature* Save time doing exhaustive literature

researches

*Wyatt: about 25,000 biomedical journals in print worldwide.

This number is rising by 4% pa - doubling time of 19 yrs

Review

Any attempt to synthesise results and conclusions of 2 or more publications on a given topic e.g. editorials, working papers

Problems Retrieval bias and publication bias

Were all studies identified? Do not see how conclusions arrived at Reviewers may be biased / conflict of interest Languages excluded?

Were all studies that met the inclusion criteria for the review identified successfully?

If not, were the results of the sample of studies included in the review representative of the results of all eligible studies?

Of controlled trials identified in five separate subjectareas within perinatal medicine, between 20 and 50 percent were identified using MEDLINE, comparedwith between 85 and 100 percent using the Oxford Database of Perinatal Trials.

Chalmers et al. 1989

Retrieval bias What can be done about it? Systematic and comprehensive search for

eligible reports

How can it be prevented? Use of structured abstracts by investigators Improvement, extension and further development

of Bibliographic retrieval systems

Explicit search criteria

Publication bias

A tendency among investigators, peer reviewers and journal editors to allow the direction and statistical significance of research findings to influence decisions regarding submission and acceptance for publication

ExamplesAnalysis of relative risks derived from studies examining the relationship between passive smoking and lung cancer suggests that studies in which no relationship has been detected may remain unpublished. Vandenbrouke, Br Med J 1988;296:391-392

However, estimates of the effect of these suggests that they would have little impact on the weight of evidence against passive smoking, 2004

A survey of authors or published reports of randomized trials revealed that between a quarter and a fifth of the trials that they have ever conducted had never been published, and that trials in which a new treatment had been found to be a superior to a standard treatment were more likely than others to have been published.

Dickersin et al.Contr Clin Trials1987;8:343-353

Publication bias

Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ  2003;326:1167-1170 (31 May), Joel Lexchin, Lisa A Bero, Benjamin Djulbegovic, Otavio Clark

Results30 studies were included (1966 – 2002). Research funded by drug companies was less likely to be published than research funded by other sources. Studies sponsored by pharmaceutical companies were more likely to have outcomes favouring the sponsor than were studies with other sponsors (odds ratio 4.05; 95% confidence interval 2.98 to 5.51; 18 comparisons). None of the 13 studies that analysed methods reported that studies funded by industry was of poorer quality.

Publication bias

What can be done about it? Grey literature Contact authors

Pitfalls of reviews

“Current medical reviews do not routinely use scientific methods to identify, assess and synthesise information.” Mulrow, 1987

50 reviews in 4 major journals, 1985-86 no statement of methods 49 summary inappropriate 47

How can the situation improve?

Better reviews (high quality, more relevant) Cochrane collaboration Systematic reviews

Improved access to reviews CCPC, Databases, Effectiveness Bulletins

Readers more skilled in making sense of reviews CASP

What is a systematic review?A review in which evidence on a topic has been

systematically identified and summarised according

to predetermined criteria.

Specific clinical questions Predefined explicit methodology Reproducible Usually review of RCTs

Meta-analysis - statistical principles

The use of statistical methods to summarise the results of independent studies into a single estimate giving more

weight to results from larger studies.

No direct comparison of patients Summary statistics are calculated for each trial Individual estimates are pooled - overall pooled estimate

(if appropriate) Gives more precise estimate of effect size

Interpretation / odds ratios

Reviews

Meta-analysis

Systematic reviews

Types of review

Why use systematic reviews?

Volume of literature Provide a basis for rational decision making Are health care effects consistent? Limit bias and reduce random error Provides more reliable results Required for ethics committees, funding agencies

Aim: To help people make well-informed decisions about health care by preparing, maintaining and promoting accessibility of systematic reviews of the effects of health care interventions.

www.cochrane.org

6 principles of Cochrane Collaboration

5000 people in 50 countries

Building on peoples’ existing enthusiasm and interests 50 Cochrane Review Groups - focus on particular areas of

health

Minimising duplication effort Avoidance of bias Keeping up to date Ensuring access

Format for a (Cochrane) review Contact Cochrane Develop a protocol Formulate the problem Locate and select studies Critical appraisal of studies

Score for selection, attrition, performance, detection, blinding biases

Collecting data Analysing and presenting results Interpreting results Improving and updating reviews

What is a Cochrane Review?http://www.cochrane.org/reviews/revstruc.htm

Cochrane Reviews investigate the effects of interventions for prevention, treatment and rehabilitation in a healthcare setting. They are designed to facilitate the choices that doctors, patients, policy makers and others face in health care. Most Cochrane Reviews are based on randomized controlled trials, but other types of evidence may also be taken into account, if appropriate. Cochrane reviews have the following general features:

General features of a Cochrane review

Structured format Detailed methods section The quality of clinical studies to be incorporated into a review is

carefully considered, using predefined criteria. A thorough and systematic search strategy, which includes

searches for unpublished and non-English records If the data collected in a review are of sufficient quality and similar

enough, they are summarised statistically in a meta-analysis, generally provides a better overall estimate of a clinical effect than the

results from individual studies. Reviews aim to be relatively easy to understand for non-experts Multinational editorial teams try to ensure that a review is applicable

in different parts of the world Reviews are updatable.

Results from newly completed or identified clinical trials can be incorporated into the review after publication. Additionally, readers can send in comments and criticisms to any review, and reviews may be changed accordingly to improve their quality.

Systematic review protocol State objectives and eligibility criteria Identify potentially eligible studies Apply eligibility criteria Refine protocol Publish protocol

Elements of a systematic review

1. Define the clinical question

2. Identify all completed studies, un/published

3. Select the studies that meet scientific validity criteria

4. Look for evidence of bias

5. Describe the scientific quality of the studies

6. Assess if quality systematically related to results

7. Describe studies with a forest plot

8. Assess if similar enough to justify combining results Calculate summary measure of effect & CI

Fletcher & Fletcher T12.1

How can we appraise a review?

How can we systematically appraise a review?

What questions should we ask?

10 questions to appraise a review Critical Appraisal Skills Programme (CASP) http://www.phru.org.uk/~casp/

Three basic types of question... Is it trustworthy? – Validity

Screening questions Detailed questions on methodology

What does it say? – Results Will it help? – Relevance

Is it trustworthy? – Validity Screening questions

Did review address a clearly focused issue? Did authors review right type of study?

Detailed questions on methodology Were all important relevant studies included? Did reviewers do enough to assess the quality

of the included studies?Randomisation system, scoring system, >1

assessor If meta-analysis performed, was it reasonable

to combine the results?

What does it say - Results

Rates Relative and absolute risk, numbers needed

to treat + confidence intervals Clinical significance Statistical significance

Will it help? – Relevance(What service providers should ask)

Is my patient... Are the interventions available to my patient... Are the outcomes relevant to my patient...

.....sufficiently different from people in the review to allow me to consider the findings inapplicable?

Design literature search for:Quantitative systematic review of randomised controlled trials comparing antibiotic with placebo for acute cough in adults.(BMJ 1998;316:906-10) Fahey T. et al.

Aim:To establish whether antibiotics are effective in

the treatment of acute cough in the community

Literature search

Medical subject headings (MeSH) Databases

Language? Study type Patients

Literature search

Medical subject headings (MeSH) Cough, bronchitis, sputum, respiratory tract

infections, chest infection Databases

Medline, EMBASE, Science Citation Index, Cochrane Controlled Trials Register

Contacted authors - ?know of unpublished trials UK drug companies ?unpublished trials

Language - not just English

Literature search Inclusion/exclusion criteria

Study type Prospective trials, formal or quasi-randomisation Placebo controlled

Patients >12 years Family practice clinic, community based o/px

dept., hospital o/px dept. Acute cough, with/out sputum, no antibiotic in

preceding week COAD excluded

Conclusions?

Treatment with antibiotic Does not affect resolution of cough or alter

course of illness Benefits

Marginal for most patients with acute cough May be outweighed by side effects (RR=1.51, CI

0.86 to 2.64)