What is a systematic review?

Systematic undertaking of a review with

a focused research question that tries to

identify, appraise, select and synthesise all

high quality research evidence relevant to

that question.

What are they used for?

To help groups and individuals make decisions to improve people’s

health. Examples include:

Recommendations and Guidelines

[e.g., United States Preventive Services Task Force (USPSTF), National

Institute for Health and Clinical Excellence (NICE).]

Should smokers routinely be advised to use quit smoking medications?

Should primary care providers routinely screen patients for

depression? What should be the components of an intervention to help

overweight children manage their weight?

Benefit Design, Coverage and Policy Decisions

[e.g., Centers for Medicare and Medicaid Services (CMS), Drug

Effectiveness Review Project (DERP), UK National Health Service

(NHS).]

Should we cover the use of medication to quit smoking?

Public Policy Would it improve the health of our community if we increase funding for mass transit and bike facilities?

Individual Patient Care Should I advise this client to use behavioral treatment in addition to medication to help her quit smoking?

Patient Decisions Should I try hypnosis to help me get over my fear of flying?

Generally

We need systematic reviews to efficiently integrate

valid information and provide a basis for rational

decision making .

Systematic reviews establish where the effects of

healthcare are consistent and where they may vary

significantly.

Producers

Cochrane

Campbell

EPPI(Evidence for Policy and Practice Information )

DARE (Database of Abstracts of Reviews of Effects )

NICE (National Institute for Health and Clinical Excellence)

Interested practitioners/ academics

Stages of a systematic review

Planning the review – i.e. identifying the need for a review, and documenting the methodology

Conducting the review – i.e. finding, selecting, appraising, extracting and synthesizing primary research studies

Reporting and dissemination – i.e. writing up and disseminating the results of the review

The systematic review process

Formulate research /

policy conclusions

Search

bibliographic

databases

Identify possible

papers from

titles/abstracts

Retrieve papers

Extract data

Further selection of

primary studies

using inclusion

criteria

Synthesis

Formulate

research

question

Design

search

strategy

Quality

appraisal

Differences Between Traditional and

Systematic Reviews

(Adapted from Cook, D. J. et. al. (1997). Ann. Intern. Med. 126: 376-380)

Feature Traditional Review Systematic Review

Question Often broad in scope Focused question

Sources &

search

Not usually specified,

potentially biased

Comprehensive sources &

explicit search strategy

Selection Rarely specified,

potentially biased

Criterion-based selection,

uniformly applied

Appraisal Variable Rigorous critical appraisal,

uniformly applied

Synthesis Often a qualitative summary Quantitative summary* when

appropriate

Inferences Sometimes evidence-based Evidence-based

*A quantitative summary that includes a statistical synthesis is a meta-

analysis

Formulating review questions

Searching & selecting studies

Study quality assessment

Data synthesis

Extracting data from studies

Steps of Doing a Systematic Review

Framing the Research Question:

PICO (T)

Evidence-based models use a process for framing a

question, locating, assessing, and evaluating

PICO (T) elements include:

Problem/Patient/Population,

Intervention/Indicator(Index),

Comparison,

Outcome,

and(optional) Time element or Type of Study.

Population: Which populations of patients are we

interested in? How can they be best described? Are there

subgroups that need to be considered?

Intervention: Which intervention, treatment or approach

should be used?

Comparators: What is/are the main alternative(s) to

compare with the intervention being considered?

Outcome: What is really important for the patient? Which

outcomes should be considered? Examples include

intermediate or short-term outcomes; mortality; morbidity

and quality of life; treatment complications; adverse effects;

rates of relapse; late morbidity and re-admission; return to

work, physical and social functioning.

Features of a well-formulated review question on the effectiveness of an intervention using the PICO framework

1. Frame the question: write out your information need in

the form of a question, for example:

Does hand washing among healthcare workers reduce

hospital acquired infections?

The question above includes the PICO elements:

PICO Mad-libs

For

P ____________ does

I ____________ compared to

C ____________ improve/reduce

O ____________ ?

What types of outcomes?

Mortality/Survival

Risk of disease

Disease free period

Quality of life

Work absenteeism

Disability/ Duration and severity of illness

Pain

16

Formulating review questions

Literature Searching & finding studies

Study quality assessment

Data synthesis

Extracting data from studies

Steps of Doing a Systematic Review

Steps of Literature Search

1) Problem formulation & Keyword extraction ◦ Specify the topic or field you want to search about &

extract its main concepts as keywords.

2) Literature search ◦ Using the keywords specified in step one, we will search

all the relevant available resources.

3) Paper evaluation ◦ Appraise the found literature & select those valid ones.

Data sources for a systematic review

Electronic Bibliographic databases ◦ MEDLINE and EMBASE

◦ The Cochrane Central Register of Controlled Trials (CENTRAL)

20

SID, Magiran, Medlib , and Irandoc

Hand searching

Checking reference lists

Look for a link to "Related" articles or “find similar

results." Also known as "snowballing."

From a relevant article, follow up on citations in the

bibliography (the "ancestry" method)

Use the "cited references" feature if available.

“Grey literature” ( thesis, Internal reports, pharmaceutical industry files)

Unpublished sources known to experts in the specialty (seek by personal communication)

Raw data from published trials

Grey literature (GL) is defined as:

"materials not published commercially or indexed by

major databases."

(http://hlwiki.slais.ubc.ca/index.php/Grey_literature)

"Fugitive," "ephemeral," "invisible" literature that may be

unpublished, unevaluated, not peer-reviewed

"Information produced on all levels of government,

academics, business and industry in electronic and print

formats not controlled by commercial publishing i.e.

where publishing is not the primary activity of the

producing body." (Luxembourg, 1997 - Expanded in

New York, 2004) (http://www.greynet.org/)

Identify Search Limits

Criteria Questions to Ask

Advise from the Cochrane Handbook for Systematic

Reviews of Interventions

Time Period

Will your review be

restricted by year of

publication, or is it

important that you cover all

years?

"Date restrictions should be applied only if it is known

that relevant studies could only have been reported

during a specific time period, for example if the

intervention was only available after a certain time

point."

Language

Should you restrict to

English language

publications only?

"Whenever possible review authors should attempt to

identify and assess for eligibility all possibly relevant

reports of trials irrespective of language of publication.

No language restrictions should be included in the

search strategy."

Publication Type Are you restricting your

search by publication type?

"Format restrictions such as excluding letters are not

recommended because letters may contain important

additional information relating to an earlier trial report

or new information about a trial not reported

elsewhere."

Geographic

Considerations

Are there any geographic

considerations to include in

your search strategy?

For example, if you were researching Chinese herbal

medicine you would need to consult Chinese literature.

Critical Appraisal tools

Critical appraisal is the systematic evaluation of clinical

research papers in order to establish:

Does this study address a clearly focused question?

Did the study use valid methods to address this question?

Are the valid results of this study important?

Are these valid, important results applicable to my patient or

population?

SYSTEMATIC REVIEW: Are the

results of the review valid?

What question (PICO) did the systematic review address?

What is best? Where do I find the information?

The main question being addressed should be clearly stated. The exposure, such as a therapy or diagnostic test, and the outcome(s) of interest will often be expressed in terms of a simple relationship.

The Title, Abstract or final paragraph of the Introduction should clearly state the question. If you still cannot ascertain what the focused question is after reading these sections, search for another paper!

This paper: Yes No Unclear

Comment:

F - Is it unlikely that important, relevant studies were missed?

What is best? Where do I find the information?

The starting point for comprehensive search for all relevant studies is the major bibliographic databases (e.g., Medline, Cochrane, EMBASE, etc) but

should also include a search of reference lists from relevant studies, and contact with experts, particularly to inquire about unpublished studies. The search should not be limited to English language only. The search strategy should include both MESH terms and text words.

The Methods section should describe the search strategy, including the terms used, in some detail. The Results section will outline the number of titles and

abstracts reviewed, the number of full-text studies retrieved, and the number of studies excluded together with the reasons for exclusion. This information may be presented in a figure or flow chart.

This paper: Yes No Unclear

Comment:

A - Were the criteria used to select articles for inclusion appropriate?

What is best? Where do I find the information?

The inclusion or exclusion of studies in a systematic review should be clearly defined a priori. The eligibility criteria used should specify the patients, interventions or exposures and outcomes of interest. In many cases the type of study design will also be a key component of the eligibility criteria.

The Methods section should describe in detail the inclusion and exclusion criteria. Normally, this will include the study design.

This paper: Yes No Unclear

Comment:

A - Were the included studies sufficiently valid for the type of question asked?

What is best? Where do I find the information?

The article should describe how the quality of each study was assessed using predetermined quality criteria appropriate to the type of clinical question (e.g., randomization, blinding and completeness of follow-up)

The Methods section should describe the assessment of quality and the criteria used. The Results section should provide information on the quality of the individual studies.

This paper: Yes No Unclear

Comment:

T - Were the results similar from study to study?

What is best? Where do I find the information?

Ideally, the results of the different studies should be similar or homogeneous. If heterogeneity exists the authors may estimate whether the differences are significant (chi-square test). Possible reasons for the heterogeneity should be explored.

The Results section should state whether the results are heterogeneous and discuss possible reasons. The forest plot should show the results of the chi-square test for heterogeneity and if discuss reasons for heterogeneity, if present.

This paper: Yes No Unclear

Comment:

What were the results?

Search Flow-chart

A Good Reference

Cochrane Handbook for Systematic

Reviews of Interventions

Available at:

http://handbook.cochrane.org/