Conducting and Conducting and Interpreting Systematic Interpreting Systematic

Reviews and Meta-Reviews and Meta-AnalysesAnalysesJuly 12, 2007July 12, 2007

Initial Notes about ReviewsInitial Notes about Reviews

Reviews are considered the “highest” form Reviews are considered the “highest” form of evidence in the EBM evidence pyramidof evidence in the EBM evidence pyramid

Techniques/issues discussed here are Techniques/issues discussed here are different from the type of review (e.g., different from the type of review (e.g., Psychological BulletinPsychological Bulletin) you are used to) you are used to

Reviews are “research of research” (this is Reviews are “research of research” (this is the derivation of the ‘meta’ component in the derivation of the ‘meta’ component in ‘meta-analysis’)‘meta-analysis’)

As such, reviews must follow, and must As such, reviews must follow, and must make explicit, their methodology just like make explicit, their methodology just like primary studies doprimary studies do

Evidence Pyramid for Treatment Effectiveness QuestionsEvidence Pyramid for Treatment Effectiveness Questions

Goal: Use best available evidenceGoal: Use best available evidence

Types of ReviewsTypes of Reviews

Systematic Review: Systematic Review: literature review focused on a literature review focused on a single question which tries to identify, appraise, select and single question which tries to identify, appraise, select and synthesize all high quality research evidence relevant to that synthesize all high quality research evidence relevant to that question question

Meta-analysis: Meta-analysis: a statistical technique often used in a statistical technique often used in systematic reviews that enables the results from a number of systematic reviews that enables the results from a number of studies to be combined into a common metric to determine the studies to be combined into a common metric to determine the average effect of a given technique. Comparisons can then be average effect of a given technique. Comparisons can then be made about the relative effectiveness of various techniques made about the relative effectiveness of various techniques

Integrative Review: Integrative Review: literature review that attempts to literature review that attempts to combine experimental and nonexperimental studies, or from combine experimental and nonexperimental studies, or from experimental and theoretical work. Can be used to review experimental and theoretical work. Can be used to review theories, evidence, or to analyze methodological issuestheories, evidence, or to analyze methodological issues

Qualitative Review: Qualitative Review: methods for combining qualitative methods for combining qualitative research studies. Techniques include meta-synthesis, formal research studies. Techniques include meta-synthesis, formal grounded theory, and meta-ethnographygrounded theory, and meta-ethnography

Review StagesReview Stages

Problem formulationProblem formulationLiterature searchLiterature searchData evaluation/extractionData evaluation/extractionData analysisData analysisPresentationPresentation

Systematic ReviewsSystematic Reviews

DefinitionDefinition: summary of the medical literature that uses : summary of the medical literature that uses explicit methods to perform a thorough literature search explicit methods to perform a thorough literature search and critical appraisal of individual studies and that uses and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid appropriate statistical techniques to combine these valid studies studies

Most evaluate the effectiveness of therapy, but systematic Most evaluate the effectiveness of therapy, but systematic reviews of diagnostic test are also importantreviews of diagnostic test are also important

Systematic reviews are:Systematic reviews are: Systematic in identification of literatureSystematic in identification of literature Explicit in statement of objectives, materials, and methodsExplicit in statement of objectives, materials, and methods Reproducible in its methodology and conclusionsReproducible in its methodology and conclusions

Best individual sources: Cochrane, PubMed Clinical QueriesBest individual sources: Cochrane, PubMed Clinical Queries

Systematic Reviews: Search Systematic Reviews: Search StrategiesStrategies

Search StrategiesSearch Strategies Electronic Databases Electronic Databases (Medline, EMBASE, Cochrane; Clinical Trials Registry (Medline, EMBASE, Cochrane; Clinical Trials Registry

Database)Database) http://ssrc.tums.ac.ir/SystematicReview/CTRDB.asphttp://ssrc.tums.ac.ir/SystematicReview/CTRDB.asp

Hand searching for published studies not in electronic Hand searching for published studies not in electronic databasesdatabases

Checking reference listsChecking reference lists Contacting authorsContacting authors

Balance between comprehensiveness and precisionBalance between comprehensiveness and precision Document search strategy in final reviewDocument search strategy in final review Additional resources:Additional resources:

Cochrane handbook: Cochrane handbook: http://www.cochrane.org/resources/handbook/index.htmhttp://www.cochrane.org/resources/handbook/index.htm

Searchable database of research papers (dissertations and Searchable database of research papers (dissertations and theses) theses) http://dissertationsandtheses.com/http://dissertationsandtheses.com/

Current Contents Connect Current Contents Connect http://http://scientific.thomson.com/products/cccscientific.thomson.com/products/ccc//

Systematic Reviews: Critical Systematic Reviews: Critical AppraisalAppraisal

Critical appraisal = quality assessmentCritical appraisal = quality assessment Is the evidence valid and relevant for your clinical question Is the evidence valid and relevant for your clinical question

or situation?or situation? Ways validity assessments are usedWays validity assessments are used

As a threshold for inclusionAs a threshold for inclusion As an explanation for differences in results between studiesAs an explanation for differences in results between studies As a sensitivity measureAs a sensitivity measure As a means for weighting studies in statistical analysesAs a means for weighting studies in statistical analyses

Dimensions of critical appraisalDimensions of critical appraisal Internal validityInternal validity ApplicabilityApplicability Precision (size of CI)Precision (size of CI)

Useful resourcesUseful resources http://ssrc.tums.ac.ir/SystematicReview/Appraisal-Tools.asphttp://ssrc.tums.ac.ir/SystematicReview/Appraisal-Tools.asp http://ssrc.tums.ac.ir/SystematicReview/RCTtools.asphttp://ssrc.tums.ac.ir/SystematicReview/RCTtools.asp

Systematic Reviews: Systematic Reviews: ConclusionsConclusions

Methodological IssuesMethodological Issues Number of studies and participantsNumber of studies and participants Size of treatment effectSize of treatment effect Precision of treatment effectPrecision of treatment effect Consistency of outcomes and resultsConsistency of outcomes and results Apparent dose-response relationships, if Apparent dose-response relationships, if

anyany Generalizability and applicability of Generalizability and applicability of

findings to other populations and settingsfindings to other populations and settings

Meta-AnalysisMeta-Analysis Meta analysis is a set of statistical techniques for combining Meta analysis is a set of statistical techniques for combining

information from different studies to derive an overall estimate of a information from different studies to derive an overall estimate of a treatment's effect. treatment's effect.

Potential issues (a “Catch 22”):Potential issues (a “Catch 22”): Publication biasPublication bias Varying quality of studiesVarying quality of studies

Suggestions for high quality meta-analyses:Suggestions for high quality meta-analyses: A formal protocol should be written specifying the exact question under A formal protocol should be written specifying the exact question under

investigation and describing the studies that will be included in the investigation and describing the studies that will be included in the analysis. analysis.

All research, not just published research, should be included. The issue of All research, not just published research, should be included. The issue of “fugitive literature” is important.“fugitive literature” is important.

Registries should be established so that studies can be tracked from their Registries should be established so that studies can be tracked from their inception and not just on publication. This idea has been given a push so inception and not just on publication. This idea has been given a push so that drug companies would not be able to publish trials showing benefit that drug companies would not be able to publish trials showing benefit from their products while suppressing those that do not. from their products while suppressing those that do not.

Measures of study quality should be includedMeasures of study quality should be included Clear eligibility criteria should be establishedClear eligibility criteria should be established Many meta analytic techniques should be used and all results should be Many meta analytic techniques should be used and all results should be

reported. A result would be considered reliable only if all of the techniques reported. A result would be considered reliable only if all of the techniques give the same result.give the same result.

Steps in Statistical AnalysisSteps in Statistical Analysis Select measure of effect (rate difference, Select measure of effect (rate difference,

odds ratio, standardized mean odds ratio, standardized mean differenceetc.)differenceetc.) RCT: rate difference (EER v. CER) or rate ratioRCT: rate difference (EER v. CER) or rate ratio Case Control: odds or rate ratiosCase Control: odds or rate ratios Standardized mean difference between E and CStandardized mean difference between E and C

Specify effects model:Specify effects model: FixedFixed: results generalizable to reviewed studies : results generalizable to reviewed studies

– assumes that there is an underlying effect – assumes that there is an underlying effect running through all studiesrunning through all studies

RandomRandom: results generalizable to universe of : results generalizable to universe of such studies (computationally more intense) – such studies (computationally more intense) – assumes that the true effect estimate varies assumes that the true effect estimate varies across studiesacross studies

Meta-Analysis: Data Analytic Meta-Analysis: Data Analytic TechniquesTechniques

Dichotomous outcomes (e.g., cured v. Dichotomous outcomes (e.g., cured v. not)not)Mantel-Haenszel: pooled odds ratio Mantel-Haenszel: pooled odds ratio

across a strata of 2x2 tablesacross a strata of 2x2 tables

Analysis example: Analysis example: http://http://www.statsdirect.com/help/chi_square_testwww.statsdirect.com/help/chi_square_tests/mh.htms/mh.htm

Meta-Analysis: Data Analytic Meta-Analysis: Data Analytic Techniques (cont’d)Techniques (cont’d)

Continuous Data: Standardized Continuous Data: Standardized difference as an example (e.g., difference as an example (e.g., Cohen’s d)Cohen’s d)d = M1-M2/SDd = M1-M2/SDPooled SD of experimental and control Pooled SD of experimental and control

groups is usually used; square root of groups is usually used; square root of the average of the two squared SD’sthe average of the two squared SD’s

Can be computed from t-test (d = 2t/Can be computed from t-test (d = 2t/dfdfOther measures: Hedges’s gOther measures: Hedges’s g

Therapeutic trials of streptokinase for MI

Van Emmerik, A.A.P., Kamphuis, J.A., Hulsbosch, A.M., & Emmelkamp, P.M.G. (2002). Single-session debriefing after psychological trauma: A meta-analysis. Lancet, 360, 766-771.

Integrative ReviewsIntegrative Reviews

Provide review of available literature Provide review of available literature according to clear methodological according to clear methodological approachapproach

Potential sources of biasPotential sources of biasIncomplete literature search stageIncomplete literature search stageErrors in data extractionErrors in data extractionInappropriate data analysis, particularly Inappropriate data analysis, particularly

in combining experimental and in combining experimental and nonexperimental methodsnonexperimental methods

Integrative ReviewsIntegrative Reviews

Defining the review question clearlyDefining the review question clearly Literature search issuesLiterature search issues

Obtaining needed literature often challengingObtaining needed literature often challenging Publication biasPublication bias Identify maximum number of primary sources Identify maximum number of primary sources

through multiple strategies; make strategies through multiple strategies; make strategies explicit (i.e., specify search terms) in reviewexplicit (i.e., specify search terms) in review

Data evaluation issuesData evaluation issues Explicit assessment of study quality – complex Explicit assessment of study quality – complex

when included studies have vastly different when included studies have vastly different designsdesigns

How to define quality of nonempirical studies?How to define quality of nonempirical studies?

Integrative Reviews (cont’d)Integrative Reviews (cont’d)

Data analysis issuesData analysis issuesDeveloping a classification system for Developing a classification system for

managing data from diverse managing data from diverse methodologiesmethodologies

Classifying studies based on some Classifying studies based on some logical, predetermined system (e.g., logical, predetermined system (e.g., type of evidence, demographic type of evidence, demographic characteristics, predetermined characteristics, predetermined conceptual classification) and analyzing conceptual classification) and analyzing each subgroup sequentiallyeach subgroup sequentially

Qualitative ReviewsQualitative Reviews Qualitative research is typically Qualitative research is typically

nonexperimental, but it may be useful to nonexperimental, but it may be useful to include in systematic reviews because it include in systematic reviews because it evaluates important aspects of clinical evaluates important aspects of clinical research, often from the patient perspective research, often from the patient perspective (e.g., “views” research)(e.g., “views” research)

Relies on inductive rather than deductive Relies on inductive rather than deductive reasoningreasoning

Qualitative research involves different Qualitative research involves different measures of report quality than do RCT’smeasures of report quality than do RCT’s

Integrating quantitative and qualitative Integrating quantitative and qualitative data sources is a key challenge to data sources is a key challenge to systematic reviewerssystematic reviewers

Types of Qualitative DataTypes of Qualitative Data

http://hsc.uwe.ac.uk/dataanalysis/qualVisDataEx.asp

Major Types of Qualitative Major Types of Qualitative ResearchResearch

Participant ObservationParticipant Observation: used to assess the impact : used to assess the impact of healthcare events from the patient perspectiveof healthcare events from the patient perspective

InterviewsInterviews: used to assess range of ideas and : used to assess range of ideas and concepts among individuals involved in healthcare concepts among individuals involved in healthcare delivery and receipt of healthcare services. Oral delivery and receipt of healthcare services. Oral histories are one examplehistories are one example Interview example: Interview example: http://http://

hsc.uwe.ac.uk/dataanalysis/qualTextDataSelf.asphsc.uwe.ac.uk/dataanalysis/qualTextDataSelf.asp Record reviewRecord review: used to assess phenomenology : used to assess phenomenology

that leads to health care decisions (e.g., what that leads to health care decisions (e.g., what factors lead to discharge or prescription of factors lead to discharge or prescription of particular medications or behavioral intervention particular medications or behavioral intervention strategies)strategies)

Focus groupsFocus groups Case studiesCase studies

Qualitative Research: Analytic Qualitative Research: Analytic Strategy ExamplesStrategy Examples

Constant Comparison MethodConstant Comparison Method: All data : All data sources are compared to all others to sources are compared to all others to search for commonalities or differencessearch for commonalities or differences““Grounded Theory” (Glaser, Strauss) – Grounded Theory” (Glaser, Strauss) –

systematic method of generating theory systematic method of generating theory from datafrom data

Ethnographic methodsEthnographic methods: How individuals : How individuals or cultural groups conceptualize or or cultural groups conceptualize or enact health/illness or health-seekingenact health/illness or health-seeking

Thomas, et al., (2004). Integrating qualitative research with trials in systematic reviews. BMJ, 328, 1010-1012.

Integrating Qualitative and Quantitative Date in Systematic Reviews