Asian Journal of Psychiatry 4 (2011) 80–85

Medical Education Corner

Practicing evidence-based psychiatry- 2. Interpreting integrative literature:Systematic reviews and meta-analyses

Julia Haj-Ibrahim, Rajiv Tandon *

Univeristy of Florida College of Medicine; Department of Psychiatry, Gainesville, FL, USA

A R T I C L E I N F O

Keywords:

Meta-analysis

Systematic review

EBM

Evidence-based

Antipsychotic

Schizophrenia

Treatment

A B S T R A C T

The practice of evidence-based medicine (EBM) requires the ability to adequately appraise relevant

clinical evidence with the intention of making an evidence-based treatment decision. Broadly, there are

three available sources of information available to the practicing clinician: individual studies, reviews of

the literature, and practice guidelines. In the first article in the series, we described the threats to validity

(T2V) approach in evaluating evidence and specifically discussed the different threats to validity in

applying the findings of a single study to an individual patient. In this article, we describe the five steps of

a systematic review and evaluate threats to validity at each of these steps. We illustrate the method by

evaluating a review relevant to the treatment question presented in a clinical vignette. We discuss

results of the meta-analytic review about comparative antipsychotic effectiveness in patients with

schizophrenia and consider application of its conclusions to the question of which antipsychotic to select

for the particular patient with schizophrenia utilizing the T2V approach.

� 2011 Published by Elsevier B.V.

Contents lists available at ScienceDirect

Asian Journal of Psychiatry

journal homepage: www.e lsev ier .com/ locate /a jp

1. Introduction

The practice of evidence-based medicine (EBM) requires theability to adequately appraise relevant evidence with the intentionof making an evidence-based treatment choice for an individualpatient. Broadly, there are three available sources of informationavailable to the practicing clinician: individual studies, reviews ofthe literature, and practice guidelines. Whereas individual studiesanalyze the association between one or more variables in apopulation of interest, literature reviews and practice guidelinessynthesize the results of many individual studies within aparticular framework. In the first article in this series (Rankupalliand Tandon, 2010), we discussed the ‘‘threats to validity’’ approach(Cook and Campbell, 2002) towards applying findings of a relevantstudy to choosing the best-possible treatment for a given patient ina specified clinical situation. In the current article, we firstsummarize types of integrative literature. We next considerthreats to the validity of conclusions derived from systematicreviews and meta-analyses. Finally, we apply an exemplar meta-analysis to answering the question presented in the clinical

* Corresponding author at: Univeristy of Florida College of Medicine, Department

of Psychiatry, 100 S. Newell Drive, Mc Knight Brain Institute, Suite L4-100,

Gainesville, FL 32611. Tel.: +352 392 3681; fax: +352 392 2579.

E-mail address: tandon@ufl.edu (R. Tandon).

1876-2018/$ – see front matter � 2011 Published by Elsevier B.V.

doi:10.1016/j.ajp.2011.02.004

vignette described in the first paper and consider potential threatsto validity (T2V) in this systematic review.

2. Integrative literature

Whereas a well-constructed, well-conducted, and properlyinterpreted relevant clinical trial provides valuable evidencetowards answering a question about ‘‘best treatment’’, there aremany reasons why it is important to evaluate the results of evensuch a trial only in the context of other trials and make an attemptto synthesize ‘‘all available relevant literature’’. There can be noperfect trial that completely addresses all threats to validity asillustrated in the first article in the series. Fortunately, there aremany good clinical trials with findings pertinent to a clinicalquestion and they might differently address the various threats tovalidity. Since different studies might provide discrepant informa-tion about the presence or absence of an association betweentreatment and outcome, whether any observed association is causal,the strength of the association (effect size), and the generalizabilityof the association, it is necessary to be able to synthesize resultsfrom multiple studies. There are different synthetic methods butthey all involve piecing together of several discrete parts of themedical literature and/or clinical experience and attempting toprovide a single ‘whole answer’ to the clinical question.

There are two broad approaches towards such synthesis:literature reviews and practice guidelines. In turn, there are anumber of statistical methods that support these approaches such

J. Haj-Ibrahim, R. Tandon / Asian Journal of Psychiatry 4 (2011) 80–85 81

as meta-analysis, decision analysis, and cost-effectiveness analy-sis. A literature overview is a summary of the medical literature ona focused clinical problem. All literature reviews require statementof the clinical question (e.g., what is the effect of treatment A onoutcome in patients with condition Y), collection of all relevantliterature, integrating findings from the different studies, summa-rizing the results, and presenting a conclusion about the effect oftreatment A on patients with condition Y. Literature reviewsemploy different degrees of rigor, however, in each of these steps.Many overviews are not comprehensive or balanced and reflect theauthors’ clinical opinions and experience. Authors of systematic

reviews utilize an explicit and structured approach towards each ofthe steps, including a precise statement of the clinical question, athorough literature search, a critical appraisal of individual studies,and a methodical integration of results from the individual studies(Cook et al., 2009). Meta-analysis is a statistical techniqueemployed to quantitatively combine and summarize findingsfrom many studies (Sacks et al., 1987; Moher et al., 1999) andsystematic reviews that employ meta-analysis as the analytictechnique to integrate findings are colloquially termed ‘‘meta-analytic reviews’’.

Clinical practice guidelines also represent an effort to synthe-size medical literature by providing explicit diagnostic andtreatment strategies for common clinical problems- the Instituteof Medicine (1992) defines them as ‘‘systematically developedstatements to assist practitioner and patient decisions aboutappropriate health care for specific clinical circumstances’’.Decision analysis and cost-effectiveness analysis are two of themany statistical approaches that can be utilized in developingclinical practice guidelines. We will review these approaches andthe application of clinical practice guidelines to the practice ofevidence-based medicine in the next paper in the series.

In this article, we focus on systematic reviews and meta-analysis. We first outline the series of steps in applying findings of asystematic review to making a treatment decision for a patient. Wethen examine the threats to validity at each of these steps thatmust be considered in the proper application of the review’sfindings to a specified clinical situation. We next provide a clinicalvignette and select a relevant meta-analytic review whoseconclusions can inform the treatment choice that needs to bemade. We illustrate the T2V approach by examining the threats tovalidity at each of the steps in applying this review’s findings to thetreatment decision required in the clinical vignette.

3. Studying a review: the threats to validity

In order to identify a relevant literature overview, one firstneeds to precisely define the clinical question that needs to beanswered; this is similar to the first step in applying a singlestudy’s findings to a treatment question (Rankupalli and Tandon,2010). In the context of a treatment decision, the question isusually in the form of ‘‘which of the following treatments A, B, C, D,. . .. will provide the greatest likelihood of the best outcome in mypatient with condition Y’’. Relevant literature reviews are thosethat provide high quality information about the cause–effect

relationship between intervention A (in comparison to interventionsB, C, D, . . .) and outcome in a population of patients with condition Y(like my patient). This intervention–comparison–outcome–popu-lation (‘ICOP’) reframing of the treatment question allows one tosearch for literature reviews that might provide relevant informa-tion. Having selected a particular systematic review, one begins thefollowing five-step process of evaluating the review’s findings andtheir application to answering the above clinical question:

1. Is the clinical problem addressed in the systematic review explicitand properly framed and is it relevant (i.e., matching ‘ICOP’)?

2. Does the systematic review incorporate all valid information byincluding a proper literature search and retrieval strategy?

3. Does the systematic review utilize a rigorous method to analyzethe data and thereby integrate available information? Meta-analysis is a specific quantitative method utilized to integratedata from multiple studies.

4. Does the systematic review provide a concise yet completepresentation of the information, a proper interpretation of theintegrated findings, and a specific conclusion?

5. Do the review’s findings and conclusions apply to the populationthat my patient is a part of?

The T2V approach is a useful tool in the examination of theseissues. As previously defined (Rankupalli and Tandon, 2010),validity is defined as the extent to which a stated conclusionapproximates the truth and is eminently reproducible (‘SCATTER’).In the context of an EBM-treatment decision based on findingsfrom a relevant literature review, it refers to the confidence withwhich the stated findings of the review may be applied to apopulation of patients ‘‘like the one I am treating’’. Correspondingto the ICOP system of framing the clinician’s treatment question,systematic reviews generally summarize their results in a similarformat: ‘‘We found that intervention A in comparison to interven-tion B resulted in this much greater improvement (effect size) orlikelihood of improvement (odds ratio) in outcome variable X in asample of patients with condition Y. Starting from the perspectivethat the stated review’s conclusions accurately summarize theeffects of the intervention on outcome, one could be partially orcompletely wrong about this conclusion because of potentialthreats to validity at each of the five steps. There are severalapproaches to minimizing or eliminating many of these threats tovalidity. These issues are briefly summarized:

3.1. Problem formulation

Although there are many similarities between individual studiesand integrative literature syntheses with regards to formulation ofthe clinical problem (i.e., both use the ICOP approach), there areimportant differences as well. Whereas concepts have to beexplicitly defined and operationalized in a research study beforeit can even begin, such precision is not obviously necessary in aliterature synthesis. Often, these concepts are clarified ‘‘as one goesalong’’ during the literature synthesis. Furthermore, since thepurpose of a literature review is to summarize results from severalstudies and individual studies differ in sample characteristics and/ormethodology, literature reviews necessarily have to allow arelatively broad definition of the clinical problem and its oper-ationalization. This is both a strength and potential drawback andtherefore presents specific validity threats which need to bemanaged. It is crucial that one assesses the fit between theconceptualization of the clinical problem and its operationalizationas also to balance the utility of combining results from a broad rangeof relevant studies while retaining the ability to evaluate howmethodological differences between them might explain differencesin findings. The following questions are useful in evaluatingpotential threats to validity at this step:

(i) Was there an explicit a priori articulation of the range ofallowed operations and clinical populations?

(ii) Was a ‘‘coding sheet’’ utilized to specify what information willbe collected from individual studies (Stock, 1994)?

3.2. Collection of all valid information

The validity of any systematic review of the medical literature isdetermined by the extent to which the entirety of the relevant

J. Haj-Ibrahim, R. Tandon / Asian Journal of Psychiatry 4 (2011) 80–8582

literature is included. A broad, ‘‘unbiased’’, and exhaustive searchof the literature for relevant studies is central to this step. Havingexplicit inclusion and exclusion criteria for ‘‘studies to be included’’before embarking on the literature search protect against potentialselection biases while concurrently defining the minimummethodological quality of relevant studies. A thorough searchstrategy, which is both comprehensive and rigorous, is essential.Multiple on-line databases, clinical trial registries, and othersources should be utilized to gather ‘‘the most complete’’ list ofpotential studies to be included. Publication bias, language bias,duplicate publication bias, citation bias, and outcome reportingbias are some of the threats to validity at this step (Egger et al.,2001). Each study thus identified should be critically evaluated asto its quality and validity of its stated findings (Rankupalli andTandon, 2010). The reliability and rigor of this data extraction stepcan be assessed by asking the following questions:

(i) Were clear inclusion and exclusion criteria for ‘‘studies to beincluded’’ specified a priori?

(ii) Was the literature search strategy comprehensive andexplicitly described?

(iii) Were potential sources of retrieval bias considered?(iv) Was there a rigorous assessment of the validity of individual

studies?(v) Were reliable methods employed for each of the elements (ii)

to (iv)?

3.3. Rigorous method of integrating data

Once all relevant data from the literature has been gathered,results of individual valid studies need to be integrated in order toprovide a singular summary finding. Since qualitative methods ofcombining data can be unreliable, meta-analysis has developed asa family of quantitative methods to provide an accurate measure ofeffect size (Khoshdel et al., 2006). Meta-analysis provides asummary estimate as a weighted mean of individual study findingsand its elements include tests of heterogeneity of study results, andestimation of a summary effect size. Meta-analysis rests on theassumption that all included studies are measuring the sameunderlying effect: for this assumption to be reasonable, the indi-vidual studies must all be of good quality and have similar design,methods, subjects, predictor and outcome variables to allow themto be combined. Many threats to the validity of a meta-analyticfinding derive from the tenuous nature of these premises(Thompson and Pocock, 1991; Jones, 2008). Secondly, whilemeta-analysis provides a powerful method to obtain a singlesummary of the effect of an intervention on an outcome fromresults of many different studies, it is important to emphasize thatonly primary research studies can establish causality whereasmeta-analytic evidence can only provide a measure of associationamong two variables. To assess potential threats to the validity ofthe meta-analytic method utilized to integrate data, the followingquestions should be asked to assess whether necessary underlyingassumptions are satisfied:

(i) Are the individual studies sufficiently similar that they can becombined?

(ii) Is the appropriate model (i.e., fixed effects versus randomeffects) utilized based on the degree of heterogeneity of studyresults?

(iii) Is the summary finding representative of the largest and mostrigorously performed studies?

(iv) Are reasons for differences in results of different studiesdiscussed?

(v) Are sufficient details about the data-analytic method andpotential validity threats clearly presented?

3.4. Proper method of summarizing and presenting data

The utility of a systematic review derives from its comprehen-sive collation of relevant data and the summary conclusion thatit provides from its analysis of these data. Transparency inpresentation of this information is therefore crucial. The authorsof the systematic review should summarize key elements ofindividual studies and ideally present this information in tabularform. In meta-analysis, the summary effect can be described as arelative risk, odds ratio, effect size, risk reduction, or numbersneeded to treat (NNT) or harm (NNH). A confidence interval aroundany such summary statistic should be provided. Meta-analysesshould also provide sensitivity analyses and an assessment of theimpact of sample size of individual studies on their observed effectwith reference to the summary statistic (e.g., funnel plot method).To assess potential threats to the validity of the summarizationand data presentation of a meta-analytic review, the followingquestions should be asked:

(i) Are the key elements of each individual study clearlydisplayed?

(ii) Are sensitivity analyses performed?(iii) Are confidence limits around the pooled effect statistic

provided?(iv) Is information about potential moderating effects on the

summary conclusions presented?

3.5. Application of findings of systematic review to

individual patient

As one next seeks to apply the findings of the meta-analyticreview to an individual patient, it is critical to evaluate whether thespectrum of subjects in the overview is similar to the specificpatient to whom conclusions of the review are to be applied. Arethe demographic characteristics and range of disease severity ofthe subjects in the studies included in the overview adequatelydescribed and match that of the patient to whom the overviewconclusions are to be applied? Does the range of interventionscovered in the studies cover the specific treatments of interestbeing considered for one’s patient? Are outcome variablesmeasured in the studies relevant? To assess potential threats tothe application of a meta-analytic review’s conclusions to anindividual patient, the following questions should be asked:

(i) Are the subjects included in the various studies adequatelydescribed and similar to ‘‘my patient’’?

(ii) Are the treatments assessed in the individual studies similar tothe ones I am considering for ‘‘my patient’’?

(iii) Are the outcomes assessed in the individual studies relevant to‘‘my patient’’?

(iv) Are potential moderators of the relationship between treat-ment and outcome adequately considered in the overview andhow might that impact the application of the overview’sconclusion to ‘‘my patient’’?

(v) Will the overview’s findings benefit the individual patient?

As one considers these various threats to generalized causalinference, it is important to appreciate the strengths andlimitations of the overview method and recognize that there canbe no perfect overview that completely avoids every possiblethreat. Consequently, replicability of any overview’s findings iscritical and complete transparency and comprehensive reportingof the methods enables replication. Applicability of an overview’sconclusions to a target population therefore involves appraisal ofthe extent to which the stated conclusion approximates the truthand is eminently reproducible (‘SCATTER’).

J. Haj-Ibrahim, R. Tandon / Asian Journal of Psychiatry 4 (2011) 80–85 83

4. Applying the T2V method to a systematic review relevant toindividual treatment decision

CASE VIGNETTE (discussed in Rankupalli and Tandon, 2010)Mr. A. is a 30-year old single male with a 3-year history of

paranoid schizophrenia. He presents to you with a 6-monthpsychotic exacerbation . . .. . .. . .. . .. . .. . .. . .. . .. . .. . .What treatmentwould you begin?

4.1. Operationalizing the question

One specific issue that needs to be addressed is selection of anantipsychotic agent. Framing the clinical question in the ICOPformat, the intervention being considered may be a particularantipsychotic agent (let us say, olanzapine), the comparatorswould include other first- and second- generation antipsychoticagents, the outcome might be freedom from psychosis or improvedsocial function, the target population would be young adults withschizophrenia, and the question would be articulated as ‘‘How doesolanzapine compare to other antipsychotic agents in reducingpsychosis and improving social function in adults with schizo-phrenia’’.

4.2. Finding a relevant overview

Utilizing the specific ICOP terms (olanzapine, antipsychotic,outcome, schizophrenia, adults, meta-analysis) to search theliterature, one finds a number of highly cited articles publishedin the previous decade including Geddes et al., 2000 (682citations), Davis et al., 2003 (656 citations), and Leucht et al.,2009 (215 citations). Authored by renowned experts in the fieldand published in the highest quality journals, the clinicianstruggles to choose the best one- is it better to go with the mostrecent one or the one with the greatest number of citations.Prudently, the clinician decides to compare the recommendationsof all three meta-analytic reviews, while paying closest attentionto the most recently published review (Leucht et al., 2009).

4.3. Evaluating the systematic review (Leucht et al., 2009)

The clinician notes that Leucht et al.’s systematic review iscomprehensive in size and scope, covering 150 studies with over20,000 participants and comparing nine second-generation anti-psychotics (SGAs) to first-generation antipsychotics (FGAs) acrossa number of efficacy and side-effect measures. The systematicreview provides a comprehensive description of the literaturesearch and retrieval strategy utilized beginning with the rigorousCochrane Schizophrenia Group register (Jadad et al., 1998; Adamset al., 2006). By including only double-blind, randomizedcontrolled clinical trials, the authors of the review sought toutilize good-quality data (Higgins and Green, 2005). Detailedinformation about studies included is provided in a comprehensiveapproximately 200-page appendix available freely for review onthe web. A rigorous meta-analytic method is utilized and clearlydescribed in the review. Finally, a broad presentation of thefindings is provided, in conjunction with a thorough description oftheir basis and includes wide-ranging sensitivity analyses. Theauthors conclude that some SGAs (notably clozapine, amisulpride,olanzapine, and risperidone in that order) but not others (notablyaripiprazole, quetiapine, sertindole, ziprasidone, and zotepine) aremore efficacious than FGAs and that SGAs differ in their side-effectprofiles without any consistent pattern of distinction from FGAs.

On the face of it, this systematic review appears to provideunambiguous and unimpeachable information about how differ-ent antipsychotics compare in the treatment of schizophrenia.Closer scrutiny of the meta-analytic review, however, reveals a

number of potential threats to validity, four of which areconsidered below:

(i) questionable assumptions about ‘‘appropriate dose ranges’’that are part of the inclusion-exclusion criteria for allowedfixed-dose studies of SGAs;

(ii) several ‘‘apples and oranges’’ problems;(iii) handling of missing data; and(iv) interpretation of data.

4.4. Definition of optimal dose ranges as part of inclusion/exclusion

criteria of allowed studies

In their discussion of inclusion-exclusion criteria for allowedstudies, the authors state ‘‘we selected only those with optimumdoses of second-generation antipsychotics as reported in dose-finding studies (amisulpride 50–300 mg per day for predominantlynegative symptoms and 400–800 mg per day for positivesymptoms, aripiprazole 10–30 mg per day, olanzapine 10–20 mgper day, quetiapine >250 mg per day, risperidone 4–6 mg per day,sertindole 16–24 mg per day, and ziprasidone 120–160 mg perday).’’ Several potential validity threats emerge. First, optimal doseranges defined here are different from those specified in variousexpert guidelines (e.g., Kane et al., 2003; Buchanan et al., 2010). Itcan be argued that there is a certain arbitrary nature to anydefinition of optimal antipsychotic dose ranges and that theauthors’ definition is as good as any, but this discrepancy isimportant to be aware of. What is more notable, however, is theabsence of any such specified dose ranges for the comparator FGAsas also the absence of defined acceptable dose ranges for clozapineand zotepine. Furthermore, amisulpride was the only agent whichhad two separate acceptable dose ranges for different kinds ofpatients (one for those with predominantly positive symptoms andone for those with predominantly negative symptoms). All of thesechoices produce potential threats to validity which need to becarefully explored.

4.5. Apples and oranges problems

One of the central assumptions of any systematic review of abody of studies is that the individual studies are sufficiently similarin their overall study design, subject characteristics, and choice ofpredictor and outcome variables to allow them to be combined.Whereas some variability in these elements is necessarilypermissible, the degree to which it occurs in some areas of thissystematic review presents significant potential threats to itsvalidity. For example, allowed study duration ranged from 3–78weeks! To consider 3-week treatment outcome data to besufficiently similar to 78-week treatment response and thereforecombinable in a single meta-analysis is a questionable assump-tion when comparing antipsychotic response in persons withschizophrenia. Similarly, a range of different measures wereutilized to define the same treatment response with varyingdegrees of arbitrariness; for example ‘‘50% reduction frombaseline in PANSS or BPRS scores, or a score of much improvedon the Clinical Global Impression Scale, were the a-priori chosencutoffs; however, when these were not available, we applied theauthors’ definitions of response’’. As discussed above, theallowance of two different optimal dose ranges for amisulpride,but no other agent, for two populations of schizophrenia patientsposes not only a potential bias but also presents an ‘‘apples andoranges’’ problem. There are a number of such potential threats tovalidity, some of which are individually examined by the authors.Several such threats are, however, not evaluated in this systematicoverview and no interactions between multiple potential threatsare assessed.

J. Haj-Ibrahim, R. Tandon / Asian Journal of Psychiatry 4 (2011) 80–8584

4.6. Management of missing data

Relevant data in a number of areas were missing from severalstudies included in this meta-analytic review. Although theauthors displayed due diligence in attempting to contact theoriginal sources for filling in these gaps, they employed a range ofdevices for interpolating unavailable data; e.g., as above ‘‘. . .whenthese were not available, we applied the authors’ definitions ofresponse’’. The impact of these decisions on the conclusions of thereview is difficult to evaluate as the extent of such datainterpolation is not described and nor is the potential threat tovalidity discussed.

4.7. Interpretation of data

The authors conclude that four SGAs (namely amisulpride,clozapine, olanzapine, and risperidone) are more effective thanFGAs with regard to both overall efficacy, and efficacy in positive,negative, and depression symptom domains (Tandon et al., 2009).In contrast, the other five SGAs (namely aripiprazole, quetiapine,sertindole, ziprasidone, and zotepine) evaluated in this reviewwere not, according to the authors, found to be more effective thanSGAs. It is noteworthy that the vast majority of included studiesutilized haloperidol as the FGA comparator and that no optimaldose range of haloperidol was defined to allow for a validcomparison. It should be noted that the most highly cited meta-analytic study of antipsychotic effectiveness in schizophrenia(Geddes et al., 2000) concluded that the dose of comparatorhaloperidol was the main determinant of any observed SGA-FGAdifference in efficacy in schizophrenia; in the meta-analysis ofLeucht et al. (2009), different average haloperidol comparatordoses were employed in the 95 included studies of different SGAs.The authors emphasize that ‘‘we did not note any consistent effectsof moderator variables, such as industry sponsorship, comparatordose, or prophylactic antiparkinsonian medication’’. The relevanceof each of these moderators has been consistently noted (Hereset al., 2006; Tandon and Nasrallah, 2006; Hugenholtz et al., 2006).Leucht and co-workers did perform several sensitivity analyses toindividually evaluate the contribution of these and some otherfactors to their results; whereas none of the factors were found tocompletely explain the authors’ findings by themselves, additiveeffects and interactions were not examined. Finally, the authorsutilized the average outcome data of individual studies eventhough biases in these studies should have necessitated adjust-ments (e.g. with CATIE, Lieberman et al., 2005; Kraemer et al.,2009; Rankupalli and Tandon, 2010).

5. Conclusion: what is the clinician to do?

Meta-analytic reviews of high quality randomized clinical trialsconstitute the highest level of evidence for treatment-relevantclinical decisions (US DHHS, 1992; Guyatt and Rennie, 2001). Whatshould the clinician do when results of different meta-analysesyield different answers to the same clinical question? How shouldthe clinician utilize results of a meta-analysis for an individualtreatment decision?

As the treating physician realized with good randomizedclinical trials (Rankupalli and Tandon, 2010), the clinician shouldalso recognize that there can be no perfect meta-analysis basedsystematic review that completely eliminates all potential threatsto validity. Again, as with randomized clinical trials (RCTs), resultsof a meta-analytic review should be considered in the context ofother evidence which includes other systematic reviews andindividual RCTs themselves.

As the EBM-practicing physician seeks scientific evidence toguide a clinical decision, one source of such evidence includes

results of a relevant meta-analytic review. In this paper, weillustrated the threats to validity (T2V) approach towardsappraising a systematic review’s findings using the example ofthe comprehensive meta-analysis by Leucht et al. (2009). Thechallenges in literal application of a review’s stated findings to aspecific clinical situation were obvious. It is equally important,however, not to consider the review as being fatally flawed and ofno value because of the existence of threats to validity. The properinterpretation of the findings has been discussed elsewhere, whichare clearly relevant to guiding antipsychotic selection for Mr. A. Atthis time, except for the greater efficacy of clozapine in treatment-refractory schizophrenia patients, differences in efficacy amongother antipsychotic agents are relatively small (Tandon et al.,2008). In contrast to relatively minor differences in efficacy,antipsychotic agents differ substantially in their side-effectprofiles. FGAs and SGAs both constitute very heterogeneousclasses of antipsychotic medications without any definitivecategorical boundary (Tandon et al., 2010).

In any event, it might be useful to examine how expert bodiesapply results of such systematic reviews in the development ofpractice guidelines. It is of interest to note that whereas the meta-analytic review might point to the choice of olanzapine as thepreferred agent for Mr. A, the most recent schizophrenia treatmentguidelines (PORT, Buchanan et al., 2010) explicitly advise againstthe choice of olanzapine for Mr. A. Practice guidelines will beconsidered in the next paper in this series.

Role of funding source

Supported by grants to RT by the National Institute of MentalHealth (MH 29118).

Contributors

All authors contributed to and have approved the finalmanuscript.

Conflict of Interest

None.

References

Adams, C.E., Coutinho, E., Davis, J.M., et al., 2006. Cochrane Schizophrenia Group:The Cochrane Library. John Wiley & Sons, Chichester.

Buchanan, R.W., Kreyenbuhl, J., Kelly, D.L., et al., 2010. The 2009 schizophreniaPORT psychopharmacological treatment recommendations and summarystatements. Schizophr. Bull. 36, 71–93.

Cook, D.J., Sackett, D.L., Spitzer, W., 2009. Methodological guidelines for systematicreviews of randomized control trials. J. Clin. Epidemiol. 48, 167–171.

Cook, T.D., Campbell, D.T., 2002. Quasi-experimentation: Design and analysis issuesfor field settings. Rand-McNally, Chicago.

Davis, J.M., Chen, N., Glick, I.D., 2003. A meta-analysis of the efficacy of second-generation antipsychotics. Arch. Gen. Psychiatry 60, 553–564.

Egger, M., Smith, G.D., Altman, D.G. (Eds.), 2001. Systematic Reviews in Healthcare:Meta-analysis in Context. BMJ Books, London.

Geddes, J., Freemantle, N., Harrison, P., Bebbington, P., 2000. Atypical antipsychoticsin the treatment of schizophrenia: systematic overview and meta-regressionanalysis. BMJ 231, 1371–1376.

Guyatt, G.H., Rennie, D., 2001. Users Guides to the Medical Literature. AMA Press,Chicago.

Heres, S., Davis, J., Maino, K., et al., 2006. Why olanzapine beats risperidone,risperidone beats quetiapine, and quetiapine beats olanzapine: an exploratoryanalysis of head-to-head comparison studies of second-generation antipsycho-tics. Am. J. Psychiatry 163, 185–194.

Higgins, J.P.T., Green, S., 2005. Cochrane Hand book for Systematic Reviews ofInterventions. John Wiley & Sons, Chichester.

Hugenholtz, G.W., Heerdink, R., Stolker, J.J., et al., 2006. Haloperidol dose when usedas active comparator in randomized controlled trials with atypical antipsycho-tics in schizophrenia: comparison with officially recommended doses. J. Clin.Psychiatry 67, 897–903.

Institute of Medicine, 1992. Guidelines for Clinical Practice: From Development toUse. National Academy Press.

J. Haj-Ibrahim, R. Tandon / Asian Journal of Psychiatry 4 (2011) 80–85 85

Jadad, A.R., Cook, D.J., Jones, A., et al., 1998. Methodology and reports of systematicreviews and meta-analyses. JAMA 280, 278–280.

Jones, D., 2008. Of medicine and meta-analysis. Nat. Rev. Drug Discov. 7, 366–367.Kane, J.M., Leucht, S., Carpenter, D., Docherty, J.P., 2003. Expert consensus guideline

series. Optimizing pharmacological treatment of psychotic disorders. Introduc-tion, methods, commentary, and summary. J. Clin. Psychiatry 64 (Suppl. 12), 5–19.

Khoshdel, A., Attia, J., Carney, S.L., 2006. Basic concepts in meta-analysis: a primerfor clinicians. Int. J. Clin. Pract. 60, 1287–1294.

Kraemer, H.C., Glick, I.D., Klein, D.F., 2009. Clinical trials design lessons from theCATIE study. Am. J. Psychiatry 166, 1222–1228.

Leucht, S., Corves, C., Arbter, D., et al., 2009. Second-generation versus first-generation antipsychotic drugs for schizophrenia: a meta-analysis. Lancet73, 31–41.

Lieberman, J.A., Stroup, S.T., McEvoy, J.P., et al., 2005. Effectiveness of antipsychoticsin patients with chronic schizophrenia. N. Engl. J. Med. 353, 1209–1223.

Moher, D., Cook, D.J., Eastwood, S., et al., 1999. Improving the quality of reports ofmeta-analyses of randomized controlled trials: the QUOROM statement. Lancet354, 1896–1900.

Rankupalli, B., Tandon, R., 2010. Practicing evidence-based psychiatry 1. Applying astudy’s findings: The threats to validity approach. Asian J. Psychiatry 3, 35–40.

Sacks, H.S., Berrier, J., Reitman, D., et al., 1987. Meta-analyses of randomized clinicaltrials. N. Engl. J. Med. 316, 45–455.

Stock, W.A., 1994. Systematic coding for research synthesis. In: Cooper, H., Hedges,L.V. (Eds.), The Handbook of Research Synthesis. Russell Sage, New York.

Tandon, R., Nasrallah, H.A., 2006. Subjecting meta-analyses to closer scrutiny: littlesupport for differential efficacy among second-generation antipsychotics. Theauthor’s reply. Arch. Gen. Psychiatry 63, 935–939.

Tandon, R., Belmaker, R.H., Gattaz, W.F., et al., 2008. World Psychiatric Associa-tion Pharmacopsychiatry Section statement on comparative effectivenessof antipsychotics in the treatment of schizophrenia. Schizophr. Res. 100,20–38.

Tandon, R., Nasrallah, H.A., Keshavan, M.S., 2009. Schizophrenia ‘‘Just the facts’’ 4.Clinical features and conceptualization. Schizophr Res. 110, 1–23.

Tandon, R., Nasrallah, H.A., Keshavan, M.S., 2010. Schizophrenia ‘‘Just the facts’’ 5.Treatment and prevention. Schizophr Res. 122, 1–23.

Thompson, S.G., Pocock, S.J., 1991. Can meta-analyses be trusted? Lancet 338,1127–1130.

US Department of Health and Human Services, Agency for Healthcare Policy andResearch, 1992. Categories of evidence relevant to specific causal relationshipsand treatments. MD. AHCPR Publications, Rockville.