A User’s Guide to Researchin Palliative Care

Systematic Reviews and Meta-Analyses

Wendy G. Anderson, M.D., M.S.,1 Megan C. McNamara, M.D., M.Sc.,2,3

and Robert M. Arnold, M.D.4–6

Introduction

You are the medical director of your local hospice.After the annual budget review, the hospice’s director

comes to you to see if there are treatments being given that arenot supported by evidence. In particular, use of supplementalhome oxygen was identified as a significant expenditure. Thedirector asks your opinion about whether supplemental oxy-gen is likely to relieve shortness of breath in patients who arenot hypoxemic. During an evidence-based medicine course,you learned that systematic reviews provide one of the stron-gest levels of evidence for guiding treatment decisions.1 Yousearch PubMed and find a systematic review and meta-analysis addressing this question.2 The review concludes ‘‘ox-ygen did not provide symptomatic benefit for cancer patientswith refractory dyspnoea who would not normally qualify forhome oxygen therapy.’’ Does this mean the hospice should stopproviding nonhypoxemic patients with home oxygen? Beforemaking a change in policy that affects so many patients, youwould like to really understand the review and its results. Howcan you evaluate the validity of this finding and determinewhether you should apply it to your patients?

The objectives of this article are to describe the methodol-ogy of systematic reviews and meta-analyses, address thecurrent status of systematic reviews and meta-analyses inpalliative care, and provide guidelines for evaluating thequality of and interpreting the results of systematic reviewsand meta-analyses. Table 1 summarizes the key concepts wewill discuss.

What are systematic reviews and meta-analyses?

A systematic review uses a defined set of methods tosummarize the scientific evidence about a particular researchquestion in a reproducible manner.3,4 Review articles can bedivided into two types: narrative and systematic. Narrativereviews do not use reproducible scientific methods to identify,assess, and synthesize information.5 Although narrative re-views can be useful in providing a particular expert’s opinionabout a topic, they have the pitfall of being subject to bias. Incontrast, a systematic review must use specific methods to

define a research question, identify, and describe relevantprimary studies, and synthesize the results, with the goal ofminimizing bias.3,6 Systematic reviews can summarize evi-dence about treatment, causation, diagnosis, or prognosis.3

Methodologically, they do this by compiling and analyzingthe results of the primary studies that address a given researchquestion. The design of these individual studies can be eitherobservational or experimental. Meta-analysis is a statisticalmethod that synthesizes the quantitative results of studies toyield a pooled estimate of the relationship between predictorsand outcomes.4 As discussed below, systematic reviewmethods must be used in order to perform a meta-analysis,but meta-analysis can only be performed in a subset of sys-tematic reviews. To perform a meta-analysis, the combinedstudies must address the same hypothesis using the samemethods.4,7

Why are systematic reviewsand meta-analyses useful?

By providing a summary of the clinical evidence about atopic, systematic reviews can help clinicians to incorporate thebest evidence into their practice. They also inform furtherresearch in the field by identifying flaws and trends in theliterature. In addition, meta-analyses can be particularlybeneficial in fields such as palliative care where primarystudies often have small sample size and thus insufficientpower to detect a statistical relationship between the predic-tors and outcomes. By combining multiple small trials, meta-analyses have the ability to show effects that were not seen inindividual, smaller studies.8

Why Should Clinicians Be Able to Critically Evaluateand Interpret the Results of Systematic Reviewsand Meta-Analyses?

It is essential that clinicians be able to critically evaluatesystematic reviews and meta-analyses to determine whetherthey should incorporate the review’s recommendations intotheir clinical practice. Although the methods used to conductsystematic reviews are aimed at decreasing bias in their

1Department of Medicine, Division of Hospital Medicine and Palliative Care Program, University of California, San Francisco, California.2Department of Medicine, Case Western Reserve University School of Medicine, Cleveland, Ohio.3Division of General Internal Medicine, Louis Stokes Cleveland Veterans Affairs Medical Center, Cleveland, Ohio.4Division of General Internal Medicine, Department of Medicine, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania.5Institute for Doctor-Patient Communication, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania.6Institute to Enhance Palliative Care, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania.

JOURNAL OF PALLIATIVE MEDICINEVolume 12, Number 10, 2009ª Mary Ann Liebert, Inc.DOI: 10.1089=jpm.2009.9954

937

results, bias can still be introduced into a systematic revieweither from the primary studies included in it or from the waythe review was conducted. Publication does not guaranteethat a review is of a high methodological quality.9 Thus,guidelines have been created to evaluate the validity of sys-tematic reviews, interpret the results, and determine how theresults should be applied in practice.3,4,6,8

Systematic Reviews and Meta-Analysesin Palliative Care

With the emergence of hospice and palliative medicine asclinical disciplines, and the emphasis on evidence-based med-icine, systematic reviews and meta-analyses are being con-ducted on topics relevant to the care of palliative care patients.For example, in the Cochrane Database of Systematic Reviews,there are currently 97 reviews registered with the Pain, Pallia-tive and Supportive Care Group.10 Systemic reviews and meta-analyses have focused on many topics related to palliative care:pain assessment and management,11–13 assessment and man-agement of other symptoms (such as dyspnea,2 nausea,14 con-stipation,15 and anxiety16), communication and patient andfamily information needs,17,18 provider education,19,20 prog-nostication,21,22 effectiveness of treatments for serious illness,23,24

and the effectiveness of palliative care from a systems perspec-tive25,26(Table 2).

What are the limitations of systematic reviewsand meta-analyses in palliative care?

While published systematic reviews and meta-analysesregarding palliative care topics have helped to summarize thecurrent evidence in the field, they have also shown the limi-tations of these methods as applied in palliative care andhighlighted areas in need of further research. These limita-tions can be categorized as definitional and methodological.From a definitional perspective, systematic reviews are lim-ited by the fact that palliative care is an emerging field andthus there is a lack of consensus about what research should

be included in systematic reviews and what should not.27 Forexample, while conducting a systematic review of end-of-lifecare and outcomes for the National Institutes of Health Stateof the Science Conference on End of-Life-Care, Lorenz et al.28

identified the need to define ‘‘end of life,’’ clarify a conceptualframework of outcomes including definitions of terms and therelationships among terms, and determine specific goals forthe review as challenges. Similarly, a 2008 review of evidencefor improving palliative care at the end of life found thatvariable literature indexing for advanced chronic illness andend of life limited the comprehensiveness of searches toidentify articles.29

From a methodological perspective, systematic reviewsand meta-analyses are limited by the quality and heteroge-neity of the existing palliative care primary literature. Sys-tematic reviews and meta-analyses have traditionally reliedon randomized controlled trials, however, few high-qualityrandomized controlled trials have been conducted in pallia-tive care.27,30 Rigorous methods for performing systematicreviews and meta-analyses on observational studies havebeen defined31; these may be particularly helpful when ap-plied to palliative care. When primary studies do exist on atopic, they are often either insufficiently powered or ofinsufficient quality to provide strong evidence to guide clin-ical care.27 Common problems related to the quality of pri-mary studies include lack of randomization or blinding inintervention studies and use of unvalidated measures foroutcomes.27,32 Finally, heterogeneity of primary studies withrespect to methods, interventions, patients, and outcomesmay limit the reviewers’ ability to make an overall assessmentof benefit or harm,27,29 resulting in conclusions that are toobroad to guide clinical care. For example, Wee et al.27 re-viewed the utility of Cochrane Reviews in guiding palliativecare practice. They identified 25 reviewed that were publishedas of December 2007, and found the quality of evidence in 23of the 25 reviews to be weak.

A systematic review that concludes that there is insufficientevidence to guide treatment decisions is still useful from bothclinical and research perspectives. From a clinical perspective,

Table 1. Key Concepts in Systematic Reviews and Meta-Analyses

& A systematic review summarizes the scientific evidence about a question of treatment, causation, diagnosis, or prognosisby using rigorous and reproducible methods to define a research question, identify and describe relevant primary studies,and synthesize the results of primary studies conducted about the topic.

& Meta-analysis is a statistical method that synthesizes the quantitative results of multiple primary studies that address thesame hypothesis using the same methods to yield a pooled estimate of the relationship between predictors and outcomes.

& Primary studies are the individual research studies that are used in the systematic review or meta-analysis.& Inclusion and exclusion criteria define which primary studies will be included in a systematic review.& Search strategies are designed to identify all the primary studies that address the research question; they include search

terms used to identify articles as well as which dates and databases the review authors search.& Meta-analysis should only be performed if the identified primary studies are similar in design and if the results of the

included studies are similar.& The results of a meta-analysis are central tendency and variability of the pooled effect of the predictor variable on the

outcome, resulting from the statistical combination of the effects from each included primary study. Central tendency is apoint estimate of the effect, and variability is the range within which the actual effect is likely to lie.

& A Forest plot visually displays the point estimate and 95% confidence interval for each study included in a meta-analysisas well as the central tendency and variability of the pooled effect.

& Bias, a systematic error in results, can be introduced into systematic reviews or meta-analyses either through flaws in theprimary studies included in the review or the methods used to conduct the review.

& Using guidelines to critically evaluate systematic reviews and meta-analyses can help clinicians assess their validity,interpret their results, and determine how to apply the results in their practice.

938 ANDERSON ET AL.

it elucidates the lack of data about a topic and directs clini-cians to made medical decisions on other grounds. For ex-ample, a Cochrane Review about medically assisted nutritionfound: ‘‘There are insufficient good quality trials to make anyrecommendations for practice with regards to the use ofmedically assisted nutrition in palliative care patients.’’23

Given that there is insufficient evidence to argue either for oragainst using medically assisted nutrition in palliative carepatients, clinicians should make decisions with patients andtheir families based on the patient’s underlying values andgoals, the physiologic risks and benefits of the treatment,and the perceived effect of the treatment on the individualpatient. From a research perspective, systematic reviews setpriorities for future research so that we can build the evi-dence in needy areas. As the amount and quality of research inpalliative care grows, systematic reviews and meta-analyseswill provide important evidence for guiding treatment deci-sions.

Finding systematic reviews and meta-analysesin palliative care

Because of the previously described limitations of system-atic reviews in palliative care, specifically lack of consensusabout indexing and lack of randomized controlled trials, it

may be necessary to look in multiple databases to identifypalliative care systematic reviews. Systematic reviews arepublished in biomedical journals and the Cochrane Databaseof Systematic Reviews. The Cochrane Database of SystematicReviews is published by The Cochrane Collaboration, an in-ternational not-for-profit organization that has the goal ofproviding independent evidence to inform clinical care.33

Reviews are conducted by Cochrane Review Groups, each ofwhich concentrates on a specific area of healthcare. Cochranereviews may be conducted with higher methodological rigorthan reviews published in journals.34 Reviews can also beidentified by searching online databases such as MEDLINE,35

EMBASE,36 CINAHL,37 and PSYCHInfo.38 MEDLINE can beaccessed through PubMed39 and OVID.40 Search strategiescan identify articles by topic and type (systematic review andmeta-analysis). For identifying articles by topic, both free-textand subject headings should be used (for example, MeSH inMEDLINE and EMTREE in EMBASE). ‘‘Palliative care’’ is aMeSH term in MEDLINE. Depending on the topic, otherMeSH terms could be considered such as ‘‘terminal care,’’‘‘critically ill,’’ ‘‘hospices,’’ ‘‘hospice care,’’ ‘‘nursing homes,’’‘‘long-term care facilities,’’ ‘‘withholding treatment,’’ and‘‘end-of-life care.’’ Systematic reviews and meta-analyses canbe identified by adding the search term ‘‘systematic review’’or ‘‘meta-analysis’’ to the topic of interest. A ‘‘SystematicReview’’ search strategy is also available in PubMed under theClinical Queries link,41 although it is broad in range so yieldsmany references that are not actually systematic reviews.42 InPubMed, there is a publication type ‘‘Meta-Analysis’’ in theAdvanced Search section. The PubMed website also includesa page of Related Sources for Searching For SystematicReviews.43

Cochrane Reviews are indexed in the databases listedabove, and can also be identified by group, topic, and keyword search on the Cochrane Library website.10 There is aPain, Palliative and Supportive Care Group as well as a Pain,Palliative, and Supportive Care Topic. The Cochrane Libraryalso includes the Database of Abstracts of Reviews of Effects(DARE),44 which contains published systematic reviewswhich are high-quality, but have not been conducted by theCochrane group nor reviewed by them.

An important factor in evaluating the utility of systematicreviews and meta-analyses once they are identified is theiryear of publication and whether and when they were lastupdated. The results and conclusion of a review may becomeinaccurate as new studies addressing the research questionare conducted and published. Cochrane Reviews have theadvantage of being updated on a regular basis. The CochraneCollaboration requires that authors update their reviews ev-ery 2 years, or, if they do not feel that an update is required,that they justify why it is not required (e.g., new data aboutthe research question are expected to be published infre-quently if at all).45 If the only systematic review availableabout a topic is more than 2 years old and has not been up-dated, we recommend that readers search the primary liter-ature about the topic, using the search strategy employed inthe review, to identify newly published studies addressing theresearch question. If no new studies are identified, or if thefindings of all newly published studies are concordant withthe results of the review, the review is likely still accurate. Ifthe findings of any newly published studies are discordantwith those of the review, the accuracy and thus clinical

Table 2. Examples of Systematic Reviews

and Meta-Analyses in Palliative Care

PainAntidepressants for neuropathic pain11

Opioid switching: a systematic and critical review12

Other symptomsOxygen for relief of dyspnoea in mildly or nonhypoxemic

patients with cancer: a systematic review andmeta-analysis2

Interventions for noisy breathing in patients near to death63

Drug therapy for anxiety in palliative care16

Communication=patient and family information needsEvidence-based recommendations for information and care

planning in cancer care17

Knowledge and information needs of informal caregivers inpalliative care: a qualitative systematic review18

Provider educationCommunication skills training for health care professionals

working with patients with cancer, their families and=orcaregivers19

A systematic review of teaching and learning in palliativecare within the medical undergraduate curriculum20

PrognosticationMeta-analysis of survival prediction with Palliative Perfor-

mance Scale22

Efficacy of treatments for serious illnessMedically assisted nutrition for palliative care in adult

patients23

Whole-brain radiotherapy for the treatment of multiple brainmetastases24

Health servicesIs there evidence that palliative care teams alter end-of-life

experiences of patients and their caregivers?25

NIH State-of-the-Science Conference Statement onimproving end-of-life care26

CRITICALLY READING SYSTEMATIC REVIEWS AND META-ANALYSES 939

applicability of the review should be questioned, especially ifthe newly published research is of higher quality (see Criti-cally Reading Systematic Reviews and Meta-Analyses below)than that used for the review.

Critically Reading Systematic Reviewsand Meta-Analyses

Once a relevant systematic review has been identified, it isessential that the reader is able to critically appraise itsmethodology. In this section we present a series of questions,adapted from the systematic review and meta-analysis liter-ature, which readers can use to evaluate the quality of sys-tematic reviews and meta-analyses (Table 3).3,4,6,45

Are the results of the review valid?

1. Does the review begin with a clearly defined clinicalquestion? The research question should be clearly ar-ticulated by the authors, including the population, in-tervention, comparison, and outcome. For example,Uronis et al.2 aimed to determine the efficacy of palli-ative oxygen, compared with air, for relief of dyspnoeain mildly or nonhypoxemic patients with cancer.

2. Were appropriate criteria defined to include and ex-clude primary studies? In a primary research study, it iscritical that the researchers define clear criteria of whichsubjects will be included and which will be excluded.Similarly, in a systematic review, the authors must de-fine explicit, reproducible criteria to define which pri-mary studies will be included in their review. Inclusionand exclusion criteria should be clearly stated, andthese criteria should be driven by the research question.It is important that these criteria be specific enough toidentify articles that address the research question, yetnot so narrow as to miss important studies. For exam-ple, in their systematic review of the evidence for im-proving palliative care at the end of life, Lorenz et al.29

specified which studies would be included: those ‘‘thataddressed ‘end of life,’ including terminal illness (forexample, advanced cancer) and chronic, eventually fatalillness with ambiguous prognosis (for example, ad-vanced dementia), and intervention studies (random-ized and nonrandomized designs) that addressed pain,dyspnea, depression, advance care planning, continu-ity, and caregiving.’’

3. Was the search strategy used likely to identify all therelevant primary studies? The goal of a thorough liter-ature search is to identify all of the literature that hasbeen published on the topic of interest. Failure to per-form the literature search in a comprehensive mannercan result in biased conclusions (see Appendix A). Thereview methods should specify how the literaturesearch was conducted in enough detail that the readercan replicate it.

4. Does the review describe the characteristics of the in-cluded primary studies? The review should give infor-mation about the design, participants, interventions,outcomes, and results for each primary study.45 Forintervention studies, methods for randomization andblinding must be provided. Additionally, the interven-tion should be described in sufficient detail so that itcan be replicated. For reviews of observational studies,

the study authors should list number and types ofstudies included (cohort, case-control, etc.), as well asthe number, demographics, and inclusion=exclusioncriteria for study participants. For intervention studies,reviewers should detail the number of participants ineach study arm. Results should be presented for eachstudy, including sample size, number of missing par-ticipants, and summary data for each group. In addi-tion, sources of funding should be identified for eachincluded study.

5. Did the primary studies that were selected for inclusionmeet the defined review eligibility criteria? This ques-tion can be answered by comparing the defined eligi-bility criteria from question #2 with the description ofthe included studies from question #4.

6. What was the methodologic quality of the primarystudies included in the review? For intervention stud-ies, quality assessment is based on randomization,blinding, withdrawal rates, and the adequacy of allo-cation concealment (see Appendix A).45 For observa-tional studies, quality assessment is directed atidentifying possible confounders and assessing themethods by which study investigators addressed theseconfounders.31,46

7. Were the primary studies similar enough in design tojustify performance of a statistical synthesis of theirresult (meta-analysis)? Meta-analysis is a statisticalmethod by which quantitative results from different

Table 3. Questions for Critically Reading

a Systematic Review3,4,6,45

I. Are the results of the review valid?1. Does the review begin with a clearly defined clinical

question?2. Were appropriate criteria defined to include and

exclude primary studies?3. Was the search strategy used likely to identify all the

relevant primary studies?4. Does the review describe the characteristics of the

included primary studies?5. Did the primary studies that were selected for

inclusion meet the defined review eligibility criteria?6. What was the methodologic quality of the primary

studies included in the review?7. Were the primary studies similar enough in design to

justify performance of a statistical synthesis of theirresult (meta-analysis)?

8. Were the results of the included studies similar?9. Are biases related to flaws in the primary studies, how

the review was conducted, or publication bias likely tohave influenced the results?

II. What are the results of the review?1. What are the results of the individual primary studies?2. If a meta-analysis was performed, what are the direc-

tion, magnitude, and precision of the pooled effect?3. What are the conclusions of the review and the strength

of the evidence for each conclusion?III. Should I use the results in my clinical practice?

1. Are the included studies representative of my practice?2. Did the review address all clinically important out-

comes?3. Could other harms or costs outweigh the potential

benefits of the review’s recommendations?

940 ANDERSON ET AL.

trials can be combined, having the advantages of (1)synthesizing the results of all the included studies toyield a summary estimate of the effect, and (2) in-creasing the sample size and thus the statistical powerof the analysis, which may potentially reveal effects thatwere not statistically significant in the individualstudies. The statistical methods of meta-analysis gobeyond simply counting the results, positive or nega-tive, for each individual study.8 The contribution ofeach primary study’s result is weighted based on itssample size and variance as well as possibly its meth-odological quality.4,47 In order to pool data, the primarystudies must have similar outcome variables. See Ap-pendix A for further detail of pooling results fromstudies with interventional compared with observa-tional designs.

8. Were the results of the included studies similar? Beforeresults of primary studies can be combined, the authorsmust first evaluate the heterogeneity of the results, orhow similar the results are from study to study, both inthe direction and magnitude of effect. This can be donevisually by graphing the point estimate of effect andconfidence intervals for each study (called a Forest plot,see further description in the ‘‘What are the results ofthe review?’’ section below) and determining whetherthe confidence intervals of the various studies overlap.If the intervals overlap, the studies results are said to bethe same or homogenous; if they do not overlap, theresults are different or heterogeneous.4 Statistical testsare also performed and reported to evaluate heteroge-neity, and the authors should specify which methodswere used to do this.4,47 If the results of the studies aresignificantly heterogeneous, the review authors shouldexplore sources of heterogeneity, which may arise fromdifferences in clinical populations, interventions, com-parisons, and outcomes. In such situations, it may bemore appropriate for authors to report results qualita-tively, rather than try to combine them quantitatively ina pooled result.48 See Appendix A for details of meth-ods used to pool results.

9. Are biases related to flaws in the primary studies, howthe review was conducted, or publication bias likely tohave influenced the results? Potential bias, which can beintroduced either through the primary studies or themethods of the review itself, is one of the most impor-tant factors for the reader to evaluate before applyingthe review results to patient care. Because observationalstudy methods are limited by inherent bias, such asconfounding and selection bias, systematic reviews that

combine the results of observational trials are them-selves subject to these biases. In contrast, interventiontrials use randomization to reduce selection and con-founding bias, and thereby increase the validity of theresults. As a result, different standards exist for evalu-ating reviews of experimental and observational tri-als.31,49 See Appendix A for details of evaluating bias inmeta-analyses.

What are the results of the review?

1. What are the results of the individual primary studies?The reviewers should present the results of each pri-mary study included in the review in detail, includingthe statistical results of the analyses performed. If theincluded studies are too heterogeneous to perform ameta-analysis, the results section of the review will endhere.

2. If a meta-analysis was performed, what are the direc-tion, magnitude, and precision of the pooled effect? Theresults of a meta-analysis are the central tendency andvariability of the pooled effect of the predictor variableon the outcome, resulting from the statistical combina-tion of the effects from each included primary study.Central tendency is a point estimate of the effect, andvariability is the range within which the true effectmight actually lie. The measure of central tendency andvariability for the pooled effect depends on whether theoutcome variable of the analysis is dichotomous (e.g.whether a patient was dead or alive at 6 months) orcontinuous (e.g., the change in pain scores of patientswho received one treatment versus another).

Dichotomous outcomes are expressed as odds ratios(OR) or relative risk (RR) (see Appendix A and Table 4).The most useful statistics for applying results to clinicalcare are the number needed to treat (NNT) and thenumber needed to harm (NNH). The NNT is defined asthe number of patients who must receive a treatmentfor one patient to benefit and the NNH is the number ofpatients who must be exposed to a risk for one to beharmed.50,51 NNT is used for treatment effects and NNHfor adverse events. The NNT is the inverse of the abso-lute risk reduction (ARR), defined as the risk in thetreatment group subtracted from the risk in the controlgroup.45 Similarly, the NNH is the inverse of the abso-lute risk increase (ARI), defined as the risk in the controlgroup subtracted from the risk in the treatment group.For example, a review of effectiveness of bisphospho-nates for relieving pain secondary to bone metastases

Table 4. Statistics Used to Describe Dichotomous Outcomes45,50,51

Statistic Abbreviation Equation

Odds n=a [number with outcome] = [number without outcome]Risk n=a [number with outcome] = [total number in group]Odds ratio OR [odds in treatment group] = [odds in control group]Relative risk RR [risk in treatment group] = [risk in control group]Absolute risk reduction or increase ARR or ARI [risk in treatment group] - [risk in control group]Number needed to

treat or harmNNT or NNH 1=ARR or 1=ARI

CRITICALLY READING SYSTEMATIC REVIEWS AND META-ANALYSES 941

reported a NNT for pain relief and a NNH for adversedrug reactions and discontinuation of therapy.52

For continuous outcomes, how pooled results areexpressed depends on whether the primary studies in-cluded used the same or different outcome measures.To pool results from different studies, results must ex-pressed in the same format.50 If the same instrumentswere used to measure the outcome in all studies, theresults can be expressed in terms of this measure, usingthe mean for central tendency and standard deviationfor variability. If different instruments are used, a stan-dardized mean difference (SMD) and 95% confidenceinterval can be computed to express the central ten-dency and variability of the combined result. For exam-ple, Uronis et al.2 used SMDs to pool results from fourstudies evaluating the efficacy of palliative oxygen forrelief of dyspnoea in mildly or nonhypoxemic patientswith cancer. The SMD can be difficult to interpret clini-cally however. Alternatively, the continuous variable

can be dichotomized (for example into patients who re-sponded had a certain degree of effect and patients whodid not), and expressed as OR or RR.45 For example, intheir review of the effectiveness of antidepressants forrelief of neuropathic pain, Sarto et al.11 reported a RR forthe percent of patients who had at least moderate painrelief.

The Forest plot is used to visually display the pointestimate of effect and 95% confidence interval for eachstudy included in the analysis, and the central tendencyand variability of the pooled effect. For example, Figure1 shows a Forest plot from Higginson and colleagues’25

systematic review and meta-analysis of the effectivenessof palliative care services. The pooled effect is shown asa diamond, the center of which is the central tendencyand the edges of which are the 95% confidence intervals.The point estimate of effect for each primary study isrepresented by a square with horizontal lines extendingfrom the square to represent the 95% confidence for the

FIG. 1. Forest plot of the effect of palliative care teams compared to conventional care on pain control. Note that the authorsperformed separate meta-analyses for each type of study: randomized controlled trial (RCT), non-RCT, and observational=retrospective. In the RCT group of studies, the confidence intervals for all the individual studies as well as the pooled effectcrosses the line of no effect (1), demonstrating no effect of palliative care teams on pain. In contrast, the confidence intervals ofall of the non-RCT studies as well as the pooled effect are <1, favoring palliative care. In the observational=retrospectivegroup of studies, the confidence intervals of 4 of the studies cross the line of no effect, yet the confidence interval of the pooledeffect does not, demonstrating that pooling results can result in a statistically significant result not evident in the individualstudies. Reprinted from Higginson IJ, Finlay IG, Goodwin DM, Hood K, Edwards AG, Cook A, Douglas HR, Normand CE: Isthere evidence that palliative care teams alter end-of-life experiences of patients and their caregivers? J Pain SymptomManage 2003;25:150–168, with permission from Elsevier.

942 ANDERSON ET AL.

study. The size of the square for each study’s point es-timate relates to the weight it was assigned in theanalysis (larger or higher quality studies typically re-ceive more weight and have larger squares).45 Thecentral vertical line on the plot is the ‘‘line of no effect.’’If the 95% confidence intervals for each primary studyor for the pooled effect cross this line then there is nostatistical association between the predictor and theoutcome. This can be either because there truly is norelationship between the predictor and outcome vari-ables, or because the sample size was too small to showthe difference.4 For dichotomous outcomes, the dia-mond and squares are OR or RRs and the line of noeffect is at 1; ORs or RRs less than 1 favor the treatmentand OR or RR greater than 1 favor the control. Forcontinuous outcomes, the diamond and squares areSMDs and the line of no effect is at 0. Negative SMDsfavor the treatment and positive SMDs favor thecontrol.45

3. What are the conclusions of the review and the strengthof the evidence for each conclusion? At the end of theanalysis, the authors should summarize their results asrelevant to clinical medicine and grade the strength ofevidence for their recommendation. Cochrane reviewsuse the Grades of Recommendation, Assessment, De-velopment and Evaluation Working Group (GRADEWorking Group) approach to rate the strength of evi-dence, which included 4 levels: high, moderate, low,and very low (Table 5).45,53 Randomized trials aregenerally considered higher strength of evidence thanobservational studies, although well-conducted obser-vational studies with sufficient sample size could pro-vide stronger evidence than poorly conducted orunderpowered randomized trials.

Should I use the results in my practice?

1. Are the included studies representative of my practice?In applying the results of systematic review to theclinical care of patients, it is critical to examine whetherone’s clinical practice is similar to the studies includedin the meta-analysis.4 Would your patients have meteligibility criteria for any of the included primarystudies? For example, a systematic review of interven-tions for the treatment of metastatic epidural spinalcord compression concluded that decompressive sur-gery was beneficial.54 However, this benefit was limited

to patients who were ambulant or had recent paraple-gia with a single area of compression and a predictedsurvival of more than 3 months. Many patients withmetastatic cancer would not meet these criteria.

2. Did the review address all important outcomes? It isalso critical to determine whether the review includedall outcomes that would be clinically relevant. For ex-ample, in studies of pain medication, the amount ofbreakthrough pain medication taken should be assessedin addition to pain scores. Patients getting a less effec-tive therapy for pain may take more breakthroughmedication and as a result have the same pain scores aspatients getting a more effective treatment; if theamount of breakthrough medication is not assessed, itwill appear that the treatments are equal in effective-ness.

3. Could other harms or costs outweigh the potentialbenefits of the review’s recommendations? Often,harms or costs associated with treatment will impacthow the results can be applied to a particular patient.For example, a systematic review concluded that lowmolecular weight heparin (LMWH) was superior tovitamin K antagonists for preventing recurrent venousthromboembolism in patients with patients.55 However,the cost of LMWH may outweigh the benefit of thetreatment for some seriously ill patients and hospices.

Summary

Systematic reviews and meta-analyses are being increas-ingly performed on topics relevant to the clinical practice ofpalliative care, and can provide a rigorous summary of theevidence about a topic as well as a pooled and thus morepowerful statistical estimate to address questions of treat-ment, causation, diagnosis, or prognosis. The utility of sys-tematic reviews and meta-analyses in palliative care iscurrently limited by a lack of consensus about which studiesand outcomes should be included, variable literature indexingcausing difficulty in identifying primary studies, and by thequality and heterogeneity of the existing palliative care pri-mary literature. As the amount and quality of research inpalliative care grows, systematic reviews and meta-analyseswill provide important evidence for guiding treatment deci-sions. Guidelines for critically evaluating systematic reviewsand meta-analyses can help clinicians assess their validity,interpret their results, and determine how to apply the resultsin their practice.

Acknowledgments

The authors would like to thank Charles B. Wessel, M.L.S.from the University of Pittsburgh Health Sciences LibrarySystem for his advice about systematic review search strate-gies.

Dr. Anderson was supported by a Junior Faculty CareerDevelopment Award from the National Palliative Care Re-search Center.

References

1. Guyatt GH, Haynes RB, Jaeschke RZ, Cook DJ, Green L,Naylor CD, Wilson MC, Richardson WS: Users’ Guides tothe Medical Literature: XXV. Evidence-based medicine:

Table 5. GRADE Working Group

Grades of Evidence53

High quality: Further research is very unlikely to changeour confidence in the estimate of effect.

Moderate quality: Further research is likely to havean important impact on our confidence in the estimateof effect and may change the estimate.

Low quality: Further research is very likely to havean important impact on our confidence in the estimateof effect and is likely to change the estimate.

Very low quality: We are very uncertain about the estimate.

CRITICALLY READING SYSTEMATIC REVIEWS AND META-ANALYSES 943

Principles for applying the Users’ Guides to patient care.Evidence-Based Medicine Working Group. JAMA 2000;284:1290–1296.

2. Uronis HE, Currow DC, McCrory DC, Samsa GP, AbernethyAP: Oxygen for relief of dyspnoea in mildly- or non-hypoxaemic patients with cancer: A systematic review andmeta-analysis. Br J Cancer 2008;98:294–299.

3. Oxman AD, Cook DJ, Guyatt GH: Users’ guides to themedical literature. VI. How to use an overview. Evidence-Based Medicine Working Group. JAMA 1994;272:1367–1371.

4. Greenhalgh T: Papers that summarise other papers(systematic reviews and meta-analyses). BMJ 1997;315:672–675.

5. Mulrow CD: The medical review article: State of the science.Ann Intern Med 1987;106:485–488.

6. Geddes J, Freemantle N, Streiner D, Reynolds S: Under-standing and interpreting systematic reviews and meta-analyses. Part 1: Rationale, search strategy, and describingresults. Evid Based Ment Health 1998;1:68–69.

7. Carlson MD, Morrison RS: User’s guide to research in pal-liative care: Why is a new series needed? J Palliat Med2008;11:1258–1261.

8. Freemantle N, Geddes J: Understanding and interpretingsystematic reviews and meta-analyses. Part 2: Meta-analyses.Evid Based Ment Health 1998;1:102–104.

9. Delaney A, Bagshaw SM, Ferland A, Manns B, LauplandKB, Doig CJ: A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care 2005;9:R575–582.

10. The Cochrane Library. www3.interscience.wiley.com=cgi-bin=mrwhome=106568753=HOME (Last accessed March 2009).

11. Saarto T, Wiffen PJ: Antidepressants for neuropathic pain.Cochrane Database Syst Rev 2007(4):CD005454.

12. Mercadante S, Bruera E: Opioid switching: A systematic andcritical review. Cancer Treat Rev 2006;32:304–315.

13. Finlay IG, Mason MD, Shelley M: Radioisotopes for thepalliation of metastatic bone cancer: A systematic review.Lancet Oncol 2005;6:392–400.

14. Glare P, Pereira G, Kristjanson LJ, Stockler M, Tattersall M:Systematic review of the efficacy of antiemetics in thetreatment of nausea in patients with far-advanced cancer.Support Care Cancer 2004;12:432–440.

15. McNicol ED, Boyce D, Schumann R, Carr DB: Mu-opioidantagonists for opioid-induced bowel dysfunction. CochraneDatabase Syst Rev 2008(2):CD006332.

16. Jackson KC, Lipman AG: Drug therapy for anxiety in pal-liative care. Cochrane Database Syst Rev 2004(1):CD004596.

17. Walling A, Lorenz KA, Dy SM, Naeim A, Sanati H, AschSM, Wenger NS: Evidence-based recommendations for in-formation and care planning in cancer care. J Clin Oncol2008;26:3896–3902.

18. Docherty A, Owens A, Asadi-Lari M, Petchey R, Williams J,Carter YH: Knowledge and information needs of informalcaregivers in palliative care: A qualitative systematic review.Palliat Med 2008;22:153–171.

19. Fellowes D, Wilkinson S, Moore P: Communication skillstraining for health care professionals working with cancerpatients, their families and=or carers. Cochrane DatabaseSyst Rev 2003(2):CD003751.

20. Lloyd-Williams M, MacLeod RD: A systematic review ofteaching and learning in palliative care within the medicalundergraduate curriculum. Med Teach 2004;26:683–690.

21. Lau F, Cloutier-Fisher D, Kuziemsky C, Black F, DowningM, Borycki E, Ho F: A systematic review of prognostic tools

for estimating survival time in palliative care. J Palliat Care2007;23:93–112.

22. Downing M, Lau F, Lesperance M, Karlson N, Shaw J,Kuziemsky C, Bernard S, Hanson L, Olajide L, Head B,Ritchie C, Harrold J, Casarett D: Meta-analysis of survivalprediction with Palliative Performance Scale. J Palliat Care2007;23:245–252; discussion 52–54.

23. Good P, Cavenagh J, Mather M, Ravenscroft P: Medicallyassisted nutrition for palliative care in adult patients. Co-chrane Database Syst Rev 2008(4):CD006274.

24. Tsao MN, Lloyd N, Wong R, Chow E, Rakovitch E, La-perriere N: Whole brain radiotherapy for the treatment ofmultiple brain metastases. Cochrane Database Syst Rev2006;3:CD003869.

25. Higginson IJ, Finlay IG, Goodwin DM, Hood K, EdwardsAG, Cook A, Douglas HR, Normand CE: Is there evidencethat palliative care teams alter end-of-life experiences ofpatients and their caregivers? J Pain Symptom Manage 2003;25:150–168.

26. National Institutes of Health State-of-the-Science ConferenceStatement. In: National Institutes of Health State-of-the-Science Conference on Improving End-of-Life Care. De-cember 6–8, 2004.

27. Wee B, Hadley G, Derry S: How useful are systematic re-views for informing palliative care practice? Survey of 25Cochrane systematic reviews. BMC Palliat Care 2008;7:13.

28. Lorenz KA, Lynn J, Morton SC, Dy SM, Shugarman LM,Wilkinson A, Mularski RA, Sun VC, Hughes RG, Rhodes SL,Maglione M, Hilton LK: Methodological approaches for asystematic review of end-of-life care. J Palliat Med 2005;8(Suppl 1):S4–11.

29. Lorenz KA, Lynn J, Dy SM, Shugarman LR, Wilkinson A,Mularski RA, Morton SC, Hughes RG, Hilton LK, MaglioneM, Rhodes SL, Rolon C, Sun VC, Shekelle PG: Evidence forimproving palliative care at the end of life: A systematicreview. Ann Intern Med 2008;148:147–159.

30. Feuer DJ, Higgins JP: Meta-analysis. Palliat Med 1999;13:433–437.

31. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD,Rennie D, Moher D, Becker BJ, Sipe TA, Thacker SB: Meta-analysis of observational studies in epidemiology: a proposalfor reporting. Meta-analysis Of Observational Studies inEpidemiology (MOOSE) group. JAMA 2000;283:2008–2012.

32. Mularski RA, Dy SM, Shugarman LR, Wilkinson AM, Lynn J,Shekelle PG, Morton SC, Sun VC, Hughes RG, Hilton LK,Maglione M, Rhodes SL, Rolon C, Lorenz KA: A systematicreview of measures of end-of-life care and its outcomes.Health Serv Res 2007;42:1848–1870.

33. The Cochrane Collaboration: www.cochrane.org=index.htm(Last accessed March 2008).)

34. Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, MoherM, Moher D: Methodology and reports of systematicreviews and meta-analyses: A comparison of Cochrane re-views with articles published in paper-based journals.JAMA 1998;280:278–280.

35. MEDLINE: National Library of Medicine. www.nlm.nih.gov=databases=databases_medline.html (Last accessed March2008).

36. EMBASE: Elsevier. www.embase.com= (Last accessedMarch 2009).

37. Cumulative Index to Nursing and Allied Health Literature:www.ebscohost.com=cinahl= (Last accessed March 2009).

38. PSYCHInfo: www.apa.org=psycinfo= (Last accessed March2008).

944 ANDERSON ET AL.

39. PubMed: www.ncbi.nlm.nih.gov=sites=entrez= (Last accessedMarch 2008).

40. OVID: www.ovid.com=site=index.jsp?top¼1 (Last accessedMarch 2008).

41. PubMed Systematic Review Search Strategy: National Li-brary of Medicine. www.ncbi.nlm.nih.gov=entrez=query=static=clinical.shtml (Last accessed March 2009).

42. The Cochrane Clinical Register of Controlled Trials: www.mrw.interscience.wiley.com=cochrane=cochrane_clcentral_articles_fs.html (Last accessed March 2009).

43. Related Sources for Searching For Systematic Reviews: www.nlm.nih.gov=bsd=pubmed_subsets=sysreviews_sources.html(Last accessed March 2009).

44. Database of Abstracts of Reviews of Effects: www.mrw.interscience.wiley.com=cochrane=cochrane_cldare_articles_fs.html (Last accessed March 2009).

45. Cochrane Collaboration: Cochrane Handbook for SystematicReviews of Interventions Version 5.0.1. 2008. www.cochrane-handbook.org (Last accessed March 2009).

46. Egger M, Schneider M, Davey Smith G: Spurious precision?Meta-analysis of observational studies. BMJ 1998;316:140–144.

47. Lau J, Ioannidis JP, Schmid CH: Quantitative synthesis insystematic reviews. Ann Intern Med 1997;127:820–826.

48. Thompson SG: Why sources of heterogeneity in meta-analysis should be investigated. BMJ 1994;309:1351–1355.

49. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, StroupDF: Improving the quality of reports of meta-analyses ofrandomised controlled trials: The QUOROM statement.Quality of Reporting of Meta-analyses. Lancet 1999;354:1896–1900.

50. Egger M, Smith GD, Phillips AN: Meta-analysis: Principlesand procedures. BMJ 1997;315:1533–1537.

51. Altman DG, Deeks JJ: Meta-analysis, Simpson’s paradox,and the number needed to treat. BMC Med Res Methodol2002;2:3.

52. Wong R, Wiffen PJ: Bisphosphonates for the relief of painsecondary to bone metastases. Cochrane Database Syst Rev2002(2):CD002068.

53. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y,Alonso-Coello P, Schunemann HJ; GRADE Working Group:GRADE: An emerging consensus on rating quality of evi-dence and strength of recommendations. BMJ 2008;336:924–926.

54. George R, Jeba J, Ramkumar G, Chacko AG, Leng M,Tharyan P: Interventions for the treatment of metastaticextradural spinal cord compression in adults. CochraneDatabase Syst Rev 2008(4):CD006716.

55. Akl EA, Barba M, Rohilla S, Terrenato I, Sperati F, Muti P,Schunemann HJ: Anticoagulation for the long term treat-ment of venous thromboembolism in patients with cancer.Cochrane Database Syst Rev 2008(2):CD006650.

56. Juni P, Holenstein F, Sterne J, Bartlett C, Egger M: Directionand impact of language bias in meta-analyses of controlledtrials: Empirical study. Int J Epidemiol 2002;31:115–123.

57. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J: Howimportant are comprehensive literature searches and theassessment of trial quality in systematic reviews? Empiricalstudy. Health Technol Assess 2003;7:1–76.

58. Cook AM, Finlay IG, Edwards AG, Hood K, Higginson IJ,Goodwin DM, Normand CE, Douglas HR: Efficiency ofsearching the grey literature in palliative care. J PainSymptom Manage 2001;22:797–801.

59. Cook DJ, Guyatt GH, Ryan G, et al.: Should unpublisheddata be included in meta-analyses? Current convictions andcontroversies. JAMA 1993;269(21):2749–53.

60. Jadad AR, Moore RA, Carroll D, Clifton J, Buckingham L,Willan A, McIlroy W, Oxman AD: Assessing the quality ofreports of randomized clinical trials: is blinding necessary?Control Clin Trials 1996;17:1–12.

61. Sterne JA, Egger M, Smith GD: Systematic reviews in healthcare: Investigating and dealing with publication and otherbiases in meta-analysis. BMJ 2001;323:101–105.

62. Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I: The caseof the misleading funnel plot. BMJ 2006;333:597–600.

63. Wee B, Hillier R: Interventions for noisy breathing in pa-tients near to death. Cochrane Database Syst Rev 2008(1):CD005177.

Address correspondence to:Wendy G. Anderson, M.D., M.S.

University of California, San Francisco521 Parnassus AvenueSuite C-126, Box 0131

San Francisco, CA 94143-0131

E-mail: anderson.wg@gmail.com

CRITICALLY READING SYSTEMATIC REVIEWS AND META-ANALYSES 945

Appendix A. Additional Detail of Methods Used in Systematic Reviews and Meta-Analyses

Bias in Systematic Reviews and Meta-Analyses

Systematic reviews are susceptible to specific types of bias,including publication bias, grey literature bias, and languagebias. Publication bias is the publication or nonpublication ofresearch findings, depending on the nature and direction ofresults. Grey literature bias refers to the fact that resultspublished in full-length journal articles are systematicallydifferent from those published in dissertations, abstracts, andconference proceedings. Language bias refers to the phe-nomenon of the publication of research findings in a partic-ular language, depending on the nature and direction ofresults. To avoid these biases, systematic reviewers shouldconduct thorough searches of medical journal databases aswell as the unpublished and non-English literature to identifypotentially relevant primary studies. Hand-searching refer-ence lists of identified articles and consulting experts in thefield are addition useful strategies for augmenting the litera-ture search. There is some controversy about the effect of in-cluding studies that are more difficult to identify on thequality of systematic reviews.56 Trials that are difficult to lo-cate are often of lower methodological quality, so includingthem in a review can actually introduce bias.57 Furthermore,searches for unpublished literature may not be fruitful. Forexample, in the course of conducting a systematic review aboutpalliative care, one group of authors found that searching forunpublished studies did not yield studies that were relevant totheir review.58 Despite these limitations, experts continue torecommend performing a thorough literature search to iden-tify all relevant studies and relying on clear inclusion andexclusion criteria and rigorous evaluation of study methodo-logical quality to exclude poor quality studies.59

Evaluating Study Quality in Intervention Studies

Scales such as the Jadad scale60 are commonly used to as-sess quality in intervention studies, though there is debateabout the utility of these scales.45 The Cochrane collaborationrecommends looking at three key characteristics (randomi-zation, blinding, and allocation concealment) and rating howwell each study’s methodology fulfills these characteristics.

Pooling Results of Primary Studies

Meta-analytic methods have been most extensively devel-oped for randomized trials. Results from observational stud-ies can also be pooled.47 However, because observationalstudies are particularly susceptible to confounding and se-lection bias, the statistical combination of data from multipleflawed studies can result in precise but inaccurate summaryestimates. As a result, some experts recommend that sys-tematic reviews of observational studies focus on examiningthe sources of differences in results between studies as op-posed to statistically combining their results.46

Two types of statistical methods are used to pool results:fixed effects models and random effects models. Whether ornot the data sets to be combined are homogenous or het-erogeneous determines which of these models should beemployed. Fixed effects models assume that the effect of thepredictor on the outcomes is the same across studies andthat all variability in effect size is due to random variation.

In contrast, random effects models assume that the effectwill vary across studies, around the overall effect.50 Thus,fixed effects models should only be employed for homoge-nous data. Random effects models are more conservative.They are appropriate for both homogenous and heteroge-neous data, but must be employed if heterogeneity is presentamong studies.4,8,47 Many meta-analyses employ bothmethods to determine whether they yield different results.Fixed and random effects models will yield a different com-bined effect only if the included studies are heterogenous.50

Methods of Assessing for Bias

Sensitivity analysis addresses bias that may have been in-troduced based on how the review was conducted. It deter-mines how dependent the magnitude and precision of thepooled effect are on the methods used in the meta-analysis.This is accomplished by altering the analysis, such as byvarying which studies are included in the analysis or com-paring results of fixed effects and random effects models, anddetermining the effect on the pooled result.4,47 Sensitivityanalysis can be particularly helpful in meta-analyses of ob-servational studies as different studies may have differentconfounders and selection biases. Robust effects should notvary dramatically when the methods of conducting theanalysis are changed slightly. Meta-regression is a techniquethat uses the study, as opposed to individual patients, as theunit of analysis. Meta-regression can be useful for evaluatingthe effect of covariates that are the same for all individuals in astudy on the pooled effect.47

Reviews of both intervention and observational studies arevulnerable to publication bias, grey literature bias, and lan-guage bias. Statistical methods exist to investigate these bia-ses.61 Funnel plots are used in some reviews to assess forpublication bias. They are scatter plots of the effect of atreatment on the horizontal axis and a measure of studyweight such as sample size on the vertical axis. Asymmetry inthis plot was originally proposed as a way to detect publica-tion bias. However, empiric evaluations of this hypothesis donot seem to support this assertion, raising concern that validresults might be discounted as a result of an asymmetricfunnel plot.62

Odds Ratios and Relative Risk

Odds are defined as the number of participants who meetthe criteria that define the experimental outcome divided bythose who do not (e.g., the number of patients who died di-vided by the number who survived). The odds ratio is theodds in the treatment group divided by the odds in the controlgroup.50 Odds are commonly used in meta-analyses for sta-tistical reasons.8,50 Risk is defined as the number of partici-pants who meet a criteria divided by the total number ofparticipants. The relative risk is the risk in the treatment groupdivided by the risk in the control group.50 The odds ratio willapproximate the relative risk for rare events, but will behigher than the relative risk for common events.50 Variabilityfor odds ratio and relative risk is expressed using 95% confi-dence intervals; there is a 95% chance that the ‘‘true’’ effect liesbetween the upper and lower bounds of this interval.45

946 ANDERSON ET AL.