ARTICLE IN PRESS

Current Anaesthesia & Critical Care (2005) 16, 391–394

0953-7112/$ - sdoi:10.1016/j.c

�CorrespondiE-mail addr

www.elsevier.com/locate/cacc

STATISTICS

Systematic reviews & meta-analyses

M.O. Columb�, A.-G. Lalkhen

Acute Intensive Care Unit, South Manchester University Hospitals Trust, Wythenshawe,Manchester M23 9LT, UK

KEYWORDSSystematic review;Meta-analysis;Evidence-based

ee front matter & 2006acc.2006.02.004

ng author.ess: columbmo@msn.co

Summary Systematic reviews and meta-analyses are becoming more prevalent inthe published literature and are routinely being used in the evidence-basedapproach to medicine. This article describes the process and methodology ofconducting systematic reviews and meta-analyses and discusses the factors thataffect the quality of the results and conclusions.& 2006 Elsevier Ltd. All rights reserved.

Definitions

Systematic review

This is the formal process of identification, apprai-sal and evaluation of primary studies and otherrelevant research to draw conclusions about aspecific issue.

Meta-analysis

This is the statistical discipline of assimilating datafrom similar smaller studies to measure an overalleffect size with improved precision. Commonly it isinvoked as part of a systematic review of theavailable literature.

Rationale

Modern medical practice demands the ‘conscien-tious, explicit and judicious use of current best

Elsevier Ltd. All rights reserv

m (M.O. Columb).

evidence in making decisions about the care ofindividual patients’.1 This ideal can easily behampered in trying to keep pace with the expo-nential growth of research literature available.Traditionally the narrative review has been used asa means of providing a summary of availableevidence to guide clinical decision-making. Narra-tive reviews however are subjective and thereforeprone to bias and error.2 The term meta-analysiswas coined in 1976 by the psychologist, Gene Glass,to describe the statistical analysis of a largecollection of results from individual studies. Hedescribed meta-analysis as the ‘analysis of ana-lyses.3 The concept of combining studies to enableconclusions to be reached had been proposed asearly as 1904 by Karl Pearson.4

Meta-analysis has advantages over narrative re-views because it estimates the size of an effect,increases power and precision, and provides arigorous framework for the appraisal of research.5,6

The difference between a systematic review and ameta-analysis is that the latter represents astatistical integration of separate studies.7 Asystematic review is an overview of primarystudies utilizing defined methods and criteria.2,7

ed.

ARTICLE IN PRESS

M.O. Columb, A.-G. Lalkhen392

Systematic reviews therefore may contain meta-analyses. The Cochrane Collaboration has as itsprimary purpose, the generation and disseminationof high-quality systematic reviews and meta-ana-lysis of medical interventions.8 The creation of thisorganization was precipitated by the exhortationsby Archie Cochrane, a physician and epidemiolo-gist. He encouraged the medical profession toprovide those interventions for which there wascollated evidence for their effectiveness.9 There-fore, systematic reviews and meta-analyses mustfollow the scientific process for any added value tobe gained.

Process

Aim

The research aim or question must be clearlydefined. This is usually to determine if a particulartreatment or intervention is beneficial. The clinicalquestion therefore needs to be precisely framedand focussed.

Methods

The protocol must define the precise eligibility andinclusion criteria for the studies to be considered.2

A rigorous and comprehensive search for allavailable primary studies must be conducted.Logical strategies include searches of electronicbibliographic databases such as Medline, PubMed,Embase and Cochrane Register of Controlled Trials(CENTRAL). Hand searching journals, proceedingsof scientific meetings, abstract publications, re-ference lists and contacting known researchers inthe field may be required.7,10–12

Quality assessment of the identified studiesshould be explicitly scored for reliability, withprospective double-blind randomized controlledtrials having the greatest weight.10,13,14 The samplesizes, baseline patient characteristics, withdrawalrates and results of primary and secondary end-points of all the studies included must be tabu-lated. The more reliable meta-analyses includeonly properly randomized controlled trials that areappropriately blinded.10,13–15

The effect of interest may be either numerical,where the data are continuous, or categorical.Meta-analysis of continuous data involves the use aweighted average of the results or differences,which implies that larger studies are given moreweight or importance. The statistical methodscomprise fixed, random, mixed effects and Baye-

sian models, the differences between these modelsbeing the manner in which the variability in theresults of different studies is explained.2 Catego-rical data, such as binary (yes/no, dead/alive), arepresented as odds ratios, numbers needed to treat,absolute and relative risk ratios.2 The precisestatistical analysis of the data must be describedto allow any interested reader to replicate theprocess.

Results

The most common graphical technique used todisplay the results of a meta-analysis is a Forestplot.2,16 The results of individual trials are orderedand presented one below the other with theirrespective 95% confidence intervals. The pooledestimate is presented at the bottom with the 95%confidence interval.2

Discussion

The Quality of Reporting of Meta-analyses (QUOR-OM) statement describes the preferred method forreporting meta-analyses in order to prevent theproduction of flawed studies.13 These recommen-dations are based on evidence that points to theproduction of unreliable meta-analyses if certaincriteria are omitted, for example the quality ofrandomized controlled trials. Studies where alloca-tion concealment was poor can lead to a 30%overestimation of the effect of an intervention.The Discussion should cover these issues so that thestrengths and limitations are identified and ap-praised. In addition, the results of sensitivityanalyses (vide infra) to test the strength of themain findings should be discussed for valid conclu-sions to be derived.

Issues

Publication biases

Significance—follows from the greater propensityfor studies with positive or statistically significantresults to be published by scientific journalscompared to negative trials.11,14

Replication—occurs when the same data arepublished in multiple articles.

Language—occurs due to failure to search forarticles other than in English, hence missingnegative trials or those with not achieving signifi-cance.

ARTICLE IN PRESS

Systematic reviews & meta-analyses 393

Selection—occurs when citations are specificallyderived from articles such as narrative reviews orexpert opinion.

Funnel plots (where the magnitude of thetreatment effect is plotted against the samplesize) may be used to detect publication biases. Asymmetrical inverted funnel implies that thestudies found are likely to be inclusive whilst anasymmetrical plot suggests that small negative orneutral studies have been omitted.7,17

Statistical heterogeneity

For studies to be combinable they should demon-strate homogeneity or similarity particularly withrespect to the subjects, pre-test variables andmethodology. Statistical tests for heterogeneityshould be performed prior to pooling data foranalysis of the effect of interest.16 Combiningheterogeneous studies may lead to irrelevant anderroneous conclusions.17

Sensitivity analysis

This involves checking to see whether alterations ofthe analyses by the omission of trials originallyincluded in the meta-analysis materially affect theoverall result. Studies may be excluded on the basisof methodological quality (excluding studies ofpoorer quality) or omitting studies stopped earlywhich are liable to bias towards rejecting the nullhypothesis.2

Controversy

Whilst the theoretical advantages of meta-analysesare clear, there are those who believe that thepractice is dubious. In a meta-analysis comparingthe effects of epidural analgesia with ropivacaineand bupivacaine on obstetric outcomes, theauthors were fortunate that basic errors in dataabstraction in five of 20 studies (with importanterrors in three) actually worked in favour of theirconclusions.18,19 A meta-analysis demonstrating thebenefit of magnesium in acute myocardial infarc-tion, later discredited by the ISIS 4 trial, is oftencited as an example by detractors.20 It has beensuggested that the incorrect conclusion reached bythe magnesium meta-analysis was probably due toselective non-publication of negative trials.17 Theassertion by the Cochrane Injuries Group that theuse of intravenous albumin in critically ill patientsresults in one death for every 17 patients washeavily criticised.21 The authors were accused of a

lack of clinical insight and of ignoring the hetero-geneity in the studies.22,23

Benefits

Meta-analyses however have also produced impor-tant contributions to clinical practice. The EarlyBreast Cancer Trialists Collaborative Group showedthat 20,000 lives per year could be saved bytamoxifen. This was accomplished by combiningdata from 55 trials.22,24 Cumulative meta-analysis,where the analysis is repeated when a new trial ispublished, may result in earlier recommendationsin terms of the clinical effectiveness of a particulartherapy. A retrospective cumulative meta-analysisfor the use of streptokinase in acute myocardialinfarction had demonstrated significant benefit by1973.10 However, a further 34,542 patients wereenrolled in studies before the treatment waslicensed after ISIS-2 in 1988.25

Conclusion

Meta-analyses that fail to produce a definitiveanswer to a clinical question can provide a potentstimulus for initiating research by highlighting theinadequacy of existing evidence.26 By indicatingthe heterogeneity of studies in terms of their end-points or outcomes, further studies may be morerobustly designed preventing potentially unethicaltrials. In the end a systematic review or meta-analysis can only be as reliable as the originalstudies.

References

1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, RichardsonWS. Evidence based medicine: what it is and what it isn’t. BrMed J 1996;312(7023):71–2.

2. Egger M, Smith GD, Phillips AN. Meta-analysis: principles andprocedures. Br Med J 1997;315(7121):1533–7.

3. Glass GV. Primary, secondary and meta-analysis of research.Educ Res 1976;5:351–79.

4. Pearson K. Report on certain enteric fever inoculationstatistics. Br Med J 1904;3:1243–6.

5. Imperial TF. Meta-analysis: when and how. Hepatology1999;29:26–31.

6. Delahaye F, Landrivon G, Ecochard R, Colin C. Meta-analysis.Health Policy 1991;19(2–3):185–96.

7. Naylor CD. Meta-analysis and the meta-epidemiology ofclinical research. Br Med J 1997;315(7109):617–9.

8. Bero L, Rennie D. The Cochrane Collaboration. Preparing,maintaining, and disseminating systematic reviews of theeffects of health care. J Am Med Assoc1995;274(24):1935–8.

9. Chalmers I, Dickersin K, Chalmers TC. Getting to grips withArchie Cochrane’s agenda. Br Med J 1992;305(6857):786–8.

ARTICLE IN PRESS

M.O. Columb, A.-G. Lalkhen394

10. Egger M, Smith GD. Meta-analysis. Potentials and promise.Br Med J 1997;315(7119):1371–4.

11. Egger M, Smith GD. Bias in location and selection of studies.Br Med J 1998;316(7124):61–6.

12. Smith GD, Egger M. Meta-analysis: unresolved issues andfurther developments. Br Med J 1996;316:221–31.

13. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF.Improving the quality of reports of meta-analyses ofrandomised controlled trials: the QUOROM statement.Quality of Reporting of Meta-analyses. Lancet1999;354(9193):1896–900.

14. Bhandari M, Devereaux PJ, Montori V, Cina C, Tandan V,Guyatt GH, et al. Users’ guide to the surgical literature: howto use a systematic literature review and meta-analysis. CanJ Surg 2004;47(1):60–7.

15. Smith GD, Egger M, Philips AN. Meta-analysis: beyond thegrand mean? Br Med J 1997;315:1610–9.

16. Greenhalgh T. How to read a paper: the basics of evidencebased medicine, 2nd ed. London: BMJ Books; 2001.

17. Egger M, Smith GD. Misleading meta-analysis. Br Med J1995;311(7007):753–4.

18. Halpern SH, Walsh V. Epidural ropivacaine versus bupiva-caine for labor: a meta-analysis. Anesth Analg 2003;96(5):1473–9.

19. Polley LS, Columb MO. Ropivacaine and bupivacaine:concentrating on dosing!. Anesth Analg 2003;96(5):1251–3.

20. ISIS-4 (Fourth International Study of Infarct Survival)Collaborative Group. ISIS-4: a randomised factorial trialassessing early oral captopril, oral mononitrate, andintravenous magnesium sulphate in 58,050 patients withsuspected acute myocardial infarction. Lancet1995;345(8951):669–85.

21. Cochrane Injuries Group Albumin Reviewers. Human albuminadministration in critically ill patients: systematic reviewof randomised controlled trials. Br Med J 1998;317(7153):235–40.

22. Horton R. The information wars. Lancet 1999;353(9148):164–5.

23. Petros A, Schindler M, Pierce C, Jacobe S, Mok Q. Humanalbumin administration in critically ill patients. Evidenceneeds to be shown in paediatrics. Br Med J1998;317(7162):882–6.

24. Early Breast Cancer Trialists’ Collaborative Group. Tamox-ifen for early breast cancer: an overview of the randomisedtrials. Lancet 1998;351(9114):1451–67.

25. ISIS-2 (Second International Study of Infarct Survival)Collaborative Group. Randomised trial of intravenousstreptokinase, oral aspirin, both, or neither among 17,187cases of suspected acute myocardial infarction: ISIS-2.Lancet 1988;2(8607):349–60.

26. Naylor D. The case for failed meta-analyses. J Eval ClinPract 1995;1:127–30.