qualitative synthesis methods and tools & challenges … · qualitative synthesis methods and...

21
Qualitative synthesis methods and tools & Challenges of SR & MA of complex interventions August 18, 2017 Mei Chung, PhD, MPH Assistant Professor Department of Public Health and Community Medicine, School of Medicine, Tufts University

Upload: tranhanh

Post on 20-Aug-2018

234 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

Qualitativesynthesismethodsandtools&ChallengesofSR&MAofcomplex

interventions

August18,2017

MeiChung,PhD,MPHAssistantProfessor

DepartmentofPublicHealthandCommunityMedicine,SchoolofMedicine,

TuftsUniversity

Page 2: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

Evidence‐basedDecisionMaking

0 Evidence‐baseddecision‐makingprocessintegrates:0 Bestavailableresearchevidence

0 Practitionerexpertiseandotheravailableresources

0 Characteristics,needs,values,andpreferencesofthosewhowillbeaffectedbytheintervention

Source:JacobsJAetal.ToolsforImplementinganEvidence‐BasedApproachinPublicHealthPractice.inPublicHealthPractice. PrevChronicDis2012;9:110324.Alsoavailableonline:http://www.cdc.gov/pcd/issues/2012/11_0324.htm [access6/26/2016]

6

Page 3: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

Qualitativevs.QuantitativeSynthesis0 Qualitativesynthesis

0 Requiredforallsystematicreviews0 Contentexpertiseisneeded0 Subjectivitycannotbetotallyavoidedsotransparencyoftheprocessisthekeytominimizebiases

0 GRADEisthemostcommonlyusedapproach(http://www.gradeworkinggroup.org/)

0 Quantitativesynthesis(e.g.,meta‐analysis)0 Empiricalandobjective0 Contentexpertiseisnotneededforpooling(butwouldhelpinterpretresults)

0 Optionalforallsystematicreviewbut..Justifications areneededforwhenmeta‐analysisCANNOTorISNOTAPPROPRIATEtobeperformed

Page 4: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

Source:Piccoli,G.B.,etal.,Lowproteindietsinpatientswithchronickidneydisease:abridgebetweenmainstreamandcomplementary‐alternativemedicines? BMCNephrol,2016.17(1):p.76.

ThisisNot aQualitativeSynthesis

8

Page 5: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

0 Definitionofquality:0 “[T]heextenttowhichallaspectsofastudy’sdesignandconductcanbeshowntoprotectagainstsystematicbias,nonsystematicbias,andinferentialerror.” (Lohr &Carey,1999)

0 Consideredtobesynonymouswithinternalvalidity0 =Whetherresultsofaprimarystudyshouldbebelieved

0 DistinctfromassessmentofQuality(riskofbias)forabodyofevidence0 =Thecertaintylevelintheconclusionsdrawnfromthesynthesisofthebodyofevidence

WhatIsQuality(RiskofBias)AssessmentofIndividualStudies?

Lohr KN, Cary TS. Jt Comm J Qual Improv 1999;25:470-9.

Page 6: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

0 SystematicErrors:0 Includeselectionbiasandconfounding,inwhichvaluestendtobeinaccurateinaparticulardirection

0 NonsystematicErrors:0 Areattributabletochance

0 InferentialErrors:0 Resultfromproblemsindataanalysisandinterpretation,suchaschoiceofthewrongstatisticalmeasureorwronglyrejectingthenullhypothesis

WhatAretheComponentsofQualityAssessment?

Lohr KN, Cary TS. Jt Comm J Qual Improv 1999;25:470-9.

Page 7: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

ConsidertheContributionofanIndividualStudytotheBodyofEvidence

Quality(systematicerrorandinferentialerror,typeofstudy)

Sizeofstudy(nonsystematicorrandomerror)

Directionandmagnitudeofresults

Relevanceofresultstokeyquestions

RiskofBias

Precision

Directness

Consistency

Applicability

Owens DK, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/60/318/2009_0805_grading.pdf.

Directvs.indirectcomparisons,healthoutcomesvs.surrogateoutcomes

Page 8: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

0 Qualityassessmentisrequiredfor:0 Interpretingresults0 Gradingthestrengthofthebodyofevidence

0 Qualityassessmentmayalsobeusedfor:0 Selectingstudiesforthereview(basedonaprioriassessmentofevidencegaps)

0 Selectingstudiesforqualitativesynthesis0 Selectingstudiesforquantitativesynthesis0 Interpretingheterogeneousfindings

ReasonsforQualityAssessment

Page 9: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

1. Classifythestudydesign2. Applypredefinedcriteriaforqualityassessmentofeachoutcomebasedon:0 Studydesign— sourcesofbiasmayvarybydesign0 Studyconduct— poorstudyconductanddiscrepancybetweendesignandconductmayincreaseriskofbias

0 Reporting— qualityassessmentmaybeinfluencedbyadequacyofreportingwheninformationonstudydesignandconductaremissing

3. Arriveatasummaryjudgmentofthestudy’squalitytoratestudiesgood,fair,orpoor0 Requiresresolutionofconflictswhentworeviewersindependentlyevaluatequality

WhatAretheStepsinQualityAssessmentofEachIndividualStudy?

HelfandM,BalshemH.In:Methodsguideforcomparativeeffectivenessreviews.Availableat:http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.

Page 10: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

Tools forRiskofBiasAssessmentofIndividualStudies

0 Manytools,butfew“validated”(rigorouslydevelopedortestedforvalidityandreliability)tools*0 Cochraneriskofbiasassessment:Interventionstudieshttp://bmg.cochrane.org/assessing‐risk‐bias‐included‐studies0 TheNewcastle‐OttawaScale:Cohortorcase‐controlstudieshttp://www.ohri.ca/programs/clinical_epidemiology/oxford.asp

*Hartling etal.ValidityandInter‐RaterReliabilityTestingofQualityAssessmentInstruments[Internet].Rockville(MD):AgencyforHealthcareResearchandQuality(US);2012Mar.Availablefromhttp://www.ncbi.nlm.nih.gov/books/NBK92293/**Salametal.Systematicreviewsonselectednutritioninterventions:descriptiveassessmentofconductandmethodologicalchallenges.BMCNutrition2015;1:9

14

Page 11: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

Many“specialized”tools

15

Page 12: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

ToolsforStrength(Quality)ofEvidenceAssessment(I)0 GRADEapproach(http://www.gradeworkinggroup.org/)

Hollands etal.Portion,packageortablewaresizeforchangingselectionandconsumptionoffood,alcoholandtobacco.CochraneDatabaseofSystematicReviews2015,Issue9.Art.No.:CD011045.

Page 13: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

ToolsforStrength(Quality)ofEvidenceAssessment(II)0 AppliedGRADEapproachwhentherewerenometa‐analysisperformed

Murad et al. Rating the certainty in evidence in the absence of a single estimate of effect. Evid Based Med. 2017 Jun;22(3):85-87.

Page 14: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

ToolsforStrengthofEvidenceAssessment(III)0 NutritionEvidenceLibrarygradingrubric:quality,quantity,consistency,impact,andgeneralizability(http://www.nel.gov/topic.cfm?cat=3385)

18

Page 15: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

ExampleofaNon‐transparentGradingProcess0 Conclusionstatement

0 Limitedevidencefromasmallnumberofstudiessuggestsalowerriskoflungcancerassociatedwithdietarypatternscontainingmorefrequentservingsofvegetables,fruits,seafood,grainsandcereals,legumesandleanvs.higherfatmeatsandlowerfatornon‐fatdairyproducts.Despitereportedmodestsignificantreductionsinrisk,definitiveconclusionscannotbeestablishedatthistimebecauseofthesmallnumberofarticles,aswellaswidevariationinstudydesign,dietaryassessmentandcaseascertainment.

0 Grade:Limited

19

Page 16: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

Howdosystematicreviewsincorporateriskofbiasassessmentsintothesynthesisofevidence?

Numberofreviews %ofcategorytotal

Wereriskofbiasassessmentsincorporatedintosynthesis?

Yes 37 63

No 20 34

Notapplicable 2 3

Howwereriskofbiasassessmentsincorporatedintosynthesis?

Sensitivityanalysis(eg,limitingtostudiesatlowestriskofbiasinasecondaryanalysis) 20 54

Narrative(discussionwithintext) 14 38

Exclusionofstudiesathighriskofbiasfrommainreviewsynthesis 5 14

Otherapproach 4 1120

Reference:Katikireddi SVetal.JEpidemiol CommunityHealth.2015Feb;69(2):189‐95. Availableathttp://jech.bmj.com/content/69/2/189

Table 4 Methods for Incorporating risk of bias assessments into reviews during synthesis

Page 17: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

MetaAnalysis0 Nonewdatabutcangeneratenewhypothesis0 Validityofmeta‐analysisresultsreliedonthequalityofthedata

Page 18: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

IntroductiontoMeta‐analysis0 Manyreadilyavailable(anduserfriendly)softwaretoallowpeoplewithoutmuchstatisticalbackgroundtoperformmeta‐analysis0 But..Mostdevelopedtocombinedrandomizedtrialresults

0 Doingameta‐analysisis(thus)veryeasybutdoingonewellishard0 Manypre‐analysissteps(e.g.,extractingquantitativedatafromtheliterature)requiresomebiostatisticsknowledge

0 Manymissingdata(intheliterature)mayberequiredtobeimputed/calculated

0 Interpretationsof(study‐level)meta‐analysisresultsaredifferentfromanalysisofpeople‐leveldata

22

Page 19: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

0 Isitreasonable?0 Arethecharacteristicsandeffectsofstudiessufficientlysimilartoestimateanaverageeffect?

0 Typesofheterogeneity:0 Clinicaldiversity:Arethestudiesofsimilartreatments,populations,settings,design,etcetera,suchthatanaverageeffectwouldbeclinicallymeaningful?

0 Methodologicaldiversity:Arethestudiesofsimilardesignandconductsuchthatanaverageeffectwouldbeclinicallymeaningful?

0 Statisticalheterogeneity:Istheobservedvariabilityofeffectsgreaterthanthatexpectedbychancealone?

Heterogeneity(Diversity):toPoolornottoPool?

23

Page 20: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

HeterogeneityIssuesaretheMainChallengesinConductingSRandMAofComplexIntervention

24Source:https://effectivehealthcare.ahrq.gov/topics/interventions‐tools‐guidance/abstract/

Page 21: Qualitative synthesis methods and tools & Challenges … · Qualitative synthesis methods and tools & ... 0 Qualitative synthesis 0 Required for all systematic reviews ... Classify

HETEROGENEOUS TREATMENT EFFECTS

IGNORE INCORPORATEESTIMATE(insensitive) EXPLAIN

FIXED EFFECTS

MODEL

DO NOT COMBINE WHEN

HETEROGENEITY IS PRESENT

RANDOM EFFECTS MODEL

SUBGROUP ANALYSES

META-REGRESSION(control rate, covariates)

DealingWithHeterogeneity:Meta‐analyticalTechniques