Systematic Reviews and

Meta-analysis

Dr Nazli Khatib (MD, PhD)Professor (Physiology)

International Initiative Impact (3ie) Evaluation AwardeeJNMC

DMIMS(DU)

1

Objectives of the talk

• To provide the needs, characteristics and constitution of

systematic review

• To provide an overview of the process of planning,

undertaking and writing a systematic review

• To provide an overview of meta-analysis

2

Levels of evidences in health care

3

The need for SR ??

• Healthcare providers, consumers & policy makers are

inundated with loads of information.

• No time, skills & resources to find, appraise and interpret this

evidence and to incorporate it into healthcare decisions.

• SR respond to this challenge by identifying, appraising and

synthesizing research-based evidence and presenting it in an

accessible format.

4

Systematic review

5

So, a systematic review.....

• ...attempts to pool all pragmatic evidences in order to answer

a specific research question.

• ...uses precise, systematic methods that are selected with a

view to minimizing bias, thus providing reliable findings from

which conclusions can be drawn and decisions made (Antman 1992,

Oxman 1993).

6

Key characteristics of an SR

• An explicit methodology

• A systematic search that identifies all studies

• An assessment of the validity of the findings of the included

studies

• A systematic presentation, and synthesis of the characteristics

and findings of the included studies.

7

Need for an SR....

SR can be conducted in an effort to:

• Resolve conflicting evidence • Help people make practical decisions about health care.• Address questions where clinical practice is uncertain• Establish feasibility of an intervention• Summarize and help people to understand the evidence. • Confirm the appropriateness of current practice or highlight a

need for future research. • Avoiding redundant unnecessary trials

8

SR with Meta-analysis

• Meta-analysis is the use of statistical methods to summarize

the results of independent studies.

• Not every SR leads to a meta-analysis. But it is better if a

meta-analysis is based on an underlying systematic review.

9

Before you start with a SR...

• Need to make sure no other systematic review identical to

yours has recently been conducted

• Need to make sure there is a need for review

10

Steps in SR

1. Define the review question

2. Search for studies

3. Select studies

4. Extract data

5. Assess risk of bias in included studies

6. Analyse data and undertake meta-analyses

7. Address reporting biases

8. Present results and ‘Summary of findings’ tables

11

SR titles

12

Scenario Structure Example Basic structure. [Intervention]

for [health problem].

Antibiotics for acute bronchitis.

Comparing two active interventions.

[Intervention A] versus [intervention B] for [health problem].

Immediate versus delayed treatment for cervical intraepithelial neoplasia.

Type of people being studied or location of intervention mentioned explicitly.

[Intervention] for [health problem] in [participant group/location].

Inhaled nitric oxide for respiratory failure in preterm infants

1. Defining the review question

• A clearly defined, focused review begins with a well framed

question.

• A well-formulated research question increase the efficiency of the

review by limiting the time and cost of identifying and obtaining

relevant literature.

• The review question should be in PICO format.

• SR can focus on broad questions, or be more narrowly defined.

13

Steps in SR

1. Define the review question

2. Search for studies

3. Select studies

4. Extract data

5. Assess risk of bias in included studies

6. Analyse data and undertake meta-analyses

7. Address reporting biases

8. Present results and ‘Summary of findings’ tables

14

2. Searching for studies

• Electronic databases

• “Grey literature” ( thesis, Internal reports, pharmaceutical industry

files)

• Checking reference lists

• Unpublished sources known to experts in the specialty (seek

by personal communication and look in trial registers)

• Hand searching

• Has to be as comprehensive as possible

15

Steps in SR

1. Define the review question

2. Search for studies

3. Select studies

4. Extract data

5. Assess risk of bias in included studies

6. Analyse data and undertake meta-analyses

7. Address reporting biases

8. Present results and ‘Summary of findings’ tables

16

3. Selecting studies(Inclusion and exclusion of studies)

• Scan titles & abstracts to retrieve possibly relevant studies

• Correspond with investigators, where appropriate, to clarify

study

• Decide if studies meet inclusion criteria and record reasons

for exclusion

• Done by two independent review authors

17

Selecting studies: Prisma chart

Steps in SR

1. Define the review question

2. Search for studies

3. Select studies

4. Extract data

5. Assess risk of bias in included studies

6. Analyse data and undertake meta-analyses

7. Address reporting biases

8. Present results and ‘Summary of findings’ tables

19

4. Extracting data

• Data needed from each study

Details of the trial

Numerical data of the outcomes in each group to enter

into meta-analysis

20

21

Steps in SR

1. Define the review question

2. Search for studies

3. Select studies

4. Extract data

5. Assess risk of bias in included studies

6. Analyse data and undertake meta-analyses

7. Address reporting biases

8. Present results and ‘Summary of findings’ tables

22

5. Assessing the risk of bias

• If poor quality trials are the building blocks of a systematic

review, the review may follow high quality methods, but the

quality of evidence may still be poor

• Use of poor quality methods may change the results of a trial

• Inclusion of poor quality studies in meta-analysis is likely to

exaggerate treatment effect

• The evaluation of the validity of the included studies is

therefore an essential.

23

Risk of bias

24

Steps in SR

1. Define the review question

2. Search for studies

3. Select studies

4. Extract data

5. Assess risk of bias in included studies

6. Analyse data and undertake meta-analyses

7. Address reporting biases

8. Present results and ‘Summary of findings’ tables

25

6. Analyse data - Meta-analysis

Principles

• Patients in one trial are NEVER directly compared with those

in another trial

• Each trial is analysed separately

• Summary statistics are collected /calculated for each trial

• Each study is weighted

• These summary statistics are added together to give the

‘meta-analysis’ 26

Why perform a meta-analysis?

• Increase power

• Improve precision of estimate

• Quantify treatment effects and their uncertainty

• Assess consistency of results

• Answer questions not posed by individual studies

• Settle controversies from conflicting studies

• Generate new hypotheses

27

When can you do a meta-analysis?

• When more than one study has estimated an effect

• When characteristics of the studies are sufficiently similar so

it makes sense to combine them

• When the data are available in the form we need to combine

them (e.g. outcome measured in similar ways)

28

When not to do a meta-analysis?

• ‘Mixing apples with oranges’

Not useful for learning about apples

Useful for learning about fruit

Meta-analysis may be meaningless and genuine effects may be

obscured if studies are too clinically diverse

• Meta-analysis in presence of serious

publication and/or reporting biases may produce an

inappropriate summary

29

Steps in doing a meta-analysis

1. Define comparisons for your review

2. Decide on outcomes for each comparison

3. Data retrieval

4. Entering and pooling data

5. Exploring heterogeneity

6. Interpreting results

30

Defining comparisons and outcome

31

Steps in doing a meta-analysis

1. Define comparisons for your review

2. Decide on outcomes for each comparison

3. Data retrieval

4. Entering and pooling data

5. Exploring heterogeneity

6. Interpreting results

32

Retrieving, entering & pooling data

33

Data analysis: Forest plot

Forest plot of comparison: Ghrelin v/s Control; Outcome: Mortality

34

Steps in SR

1. Define the review question

2. Search for studies

3. Select studies

4. Extract data

5. Assess risk of bias in included studies

6. Analyse data and undertake meta-analyses

7. Address reporting biases

8. Present results and ‘Summary of findings’ tables

35

8. Summary of findings table (SoF)

• There are 3 key parts of a SoF table:

Information about the review

Summary of the statistical results

Grade of the quality of evidence.

36

Composition of a SR

37

Grants for SR

• ICMR

• 3ie’s Systematic Review Call 5 is funding 8 systematic reviews.

• Department For International Development, UK Aid are

commissioning 10 new SR on topics such as education, health,

infrastructure and humanitarian assistance.

• NIHR Cochrane Programme Grant Scheme

• NIHR Cochrane Incentive Awards

• USAID funded reviews38

Grants for SR

• 3ie, along with the Canadian International Development

Agency (CIDA), the United States Agency for International

Development (USAID), and the Millennium Challenge

Corporation (MCC) is inviting proposals for SR in international

development. This call will fund up to 15 review questions in

the areas of agriculture, infrastructure, business

development, governance and education interventions.

39

Summary

• SR is “the application of scientific strategies that limit bias by

the systematic assembly, critical appraisal and synthesis of all

relevant studies on a specific topic.” (Cook 1995)

• Help people to resolve conflicting evidence & make practical

decisions about health care.

• Steps in SR: Def RQ Search studies Select studies

Extract data Assess risk of bias Analyse data Address

reporting biases Present results and ‘SoF’ tables Interpret

results and draw conclusions 40

Summary...

• Meta-analysis is the use of statistical methods to summarize

the results of independent studies.

• Steps in meta-analysis: Define comparisons & outcomes

Data retrieval Entering and pooling data Exploring

heterogeneity Interpreting results

41

To increase the power & precision

42

Generate new hypotheses

43

44