qualitative synthesis methods and tools & challenges … · qualitative synthesis methods and...
TRANSCRIPT
Qualitativesynthesismethodsandtools&ChallengesofSR&MAofcomplex
interventions
August18,2017
MeiChung,PhD,MPHAssistantProfessor
DepartmentofPublicHealthandCommunityMedicine,SchoolofMedicine,
TuftsUniversity
Evidence‐basedDecisionMaking
0 Evidence‐baseddecision‐makingprocessintegrates:0 Bestavailableresearchevidence
0 Practitionerexpertiseandotheravailableresources
0 Characteristics,needs,values,andpreferencesofthosewhowillbeaffectedbytheintervention
Source:JacobsJAetal.ToolsforImplementinganEvidence‐BasedApproachinPublicHealthPractice.inPublicHealthPractice. PrevChronicDis2012;9:110324.Alsoavailableonline:http://www.cdc.gov/pcd/issues/2012/11_0324.htm [access6/26/2016]
6
Qualitativevs.QuantitativeSynthesis0 Qualitativesynthesis
0 Requiredforallsystematicreviews0 Contentexpertiseisneeded0 Subjectivitycannotbetotallyavoidedsotransparencyoftheprocessisthekeytominimizebiases
0 GRADEisthemostcommonlyusedapproach(http://www.gradeworkinggroup.org/)
0 Quantitativesynthesis(e.g.,meta‐analysis)0 Empiricalandobjective0 Contentexpertiseisnotneededforpooling(butwouldhelpinterpretresults)
0 Optionalforallsystematicreviewbut..Justifications areneededforwhenmeta‐analysisCANNOTorISNOTAPPROPRIATEtobeperformed
Source:Piccoli,G.B.,etal.,Lowproteindietsinpatientswithchronickidneydisease:abridgebetweenmainstreamandcomplementary‐alternativemedicines? BMCNephrol,2016.17(1):p.76.
ThisisNot aQualitativeSynthesis
8
0 Definitionofquality:0 “[T]heextenttowhichallaspectsofastudy’sdesignandconductcanbeshowntoprotectagainstsystematicbias,nonsystematicbias,andinferentialerror.” (Lohr &Carey,1999)
0 Consideredtobesynonymouswithinternalvalidity0 =Whetherresultsofaprimarystudyshouldbebelieved
0 DistinctfromassessmentofQuality(riskofbias)forabodyofevidence0 =Thecertaintylevelintheconclusionsdrawnfromthesynthesisofthebodyofevidence
WhatIsQuality(RiskofBias)AssessmentofIndividualStudies?
Lohr KN, Cary TS. Jt Comm J Qual Improv 1999;25:470-9.
0 SystematicErrors:0 Includeselectionbiasandconfounding,inwhichvaluestendtobeinaccurateinaparticulardirection
0 NonsystematicErrors:0 Areattributabletochance
0 InferentialErrors:0 Resultfromproblemsindataanalysisandinterpretation,suchaschoiceofthewrongstatisticalmeasureorwronglyrejectingthenullhypothesis
WhatAretheComponentsofQualityAssessment?
Lohr KN, Cary TS. Jt Comm J Qual Improv 1999;25:470-9.
ConsidertheContributionofanIndividualStudytotheBodyofEvidence
Quality(systematicerrorandinferentialerror,typeofstudy)
Sizeofstudy(nonsystematicorrandomerror)
Directionandmagnitudeofresults
Relevanceofresultstokeyquestions
RiskofBias
Precision
Directness
Consistency
Applicability
Owens DK, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/60/318/2009_0805_grading.pdf.
Directvs.indirectcomparisons,healthoutcomesvs.surrogateoutcomes
0 Qualityassessmentisrequiredfor:0 Interpretingresults0 Gradingthestrengthofthebodyofevidence
0 Qualityassessmentmayalsobeusedfor:0 Selectingstudiesforthereview(basedonaprioriassessmentofevidencegaps)
0 Selectingstudiesforqualitativesynthesis0 Selectingstudiesforquantitativesynthesis0 Interpretingheterogeneousfindings
ReasonsforQualityAssessment
1. Classifythestudydesign2. Applypredefinedcriteriaforqualityassessmentofeachoutcomebasedon:0 Studydesign— sourcesofbiasmayvarybydesign0 Studyconduct— poorstudyconductanddiscrepancybetweendesignandconductmayincreaseriskofbias
0 Reporting— qualityassessmentmaybeinfluencedbyadequacyofreportingwheninformationonstudydesignandconductaremissing
3. Arriveatasummaryjudgmentofthestudy’squalitytoratestudiesgood,fair,orpoor0 Requiresresolutionofconflictswhentworeviewersindependentlyevaluatequality
WhatAretheStepsinQualityAssessmentofEachIndividualStudy?
HelfandM,BalshemH.In:Methodsguideforcomparativeeffectivenessreviews.Availableat:http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
Tools forRiskofBiasAssessmentofIndividualStudies
0 Manytools,butfew“validated”(rigorouslydevelopedortestedforvalidityandreliability)tools*0 Cochraneriskofbiasassessment:Interventionstudieshttp://bmg.cochrane.org/assessing‐risk‐bias‐included‐studies0 TheNewcastle‐OttawaScale:Cohortorcase‐controlstudieshttp://www.ohri.ca/programs/clinical_epidemiology/oxford.asp
*Hartling etal.ValidityandInter‐RaterReliabilityTestingofQualityAssessmentInstruments[Internet].Rockville(MD):AgencyforHealthcareResearchandQuality(US);2012Mar.Availablefromhttp://www.ncbi.nlm.nih.gov/books/NBK92293/**Salametal.Systematicreviewsonselectednutritioninterventions:descriptiveassessmentofconductandmethodologicalchallenges.BMCNutrition2015;1:9
14
Many“specialized”tools
15
ToolsforStrength(Quality)ofEvidenceAssessment(I)0 GRADEapproach(http://www.gradeworkinggroup.org/)
Hollands etal.Portion,packageortablewaresizeforchangingselectionandconsumptionoffood,alcoholandtobacco.CochraneDatabaseofSystematicReviews2015,Issue9.Art.No.:CD011045.
ToolsforStrength(Quality)ofEvidenceAssessment(II)0 AppliedGRADEapproachwhentherewerenometa‐analysisperformed
Murad et al. Rating the certainty in evidence in the absence of a single estimate of effect. Evid Based Med. 2017 Jun;22(3):85-87.
ToolsforStrengthofEvidenceAssessment(III)0 NutritionEvidenceLibrarygradingrubric:quality,quantity,consistency,impact,andgeneralizability(http://www.nel.gov/topic.cfm?cat=3385)
18
ExampleofaNon‐transparentGradingProcess0 Conclusionstatement
0 Limitedevidencefromasmallnumberofstudiessuggestsalowerriskoflungcancerassociatedwithdietarypatternscontainingmorefrequentservingsofvegetables,fruits,seafood,grainsandcereals,legumesandleanvs.higherfatmeatsandlowerfatornon‐fatdairyproducts.Despitereportedmodestsignificantreductionsinrisk,definitiveconclusionscannotbeestablishedatthistimebecauseofthesmallnumberofarticles,aswellaswidevariationinstudydesign,dietaryassessmentandcaseascertainment.
0 Grade:Limited
19
Howdosystematicreviewsincorporateriskofbiasassessmentsintothesynthesisofevidence?
Numberofreviews %ofcategorytotal
Wereriskofbiasassessmentsincorporatedintosynthesis?
Yes 37 63
No 20 34
Notapplicable 2 3
Howwereriskofbiasassessmentsincorporatedintosynthesis?
Sensitivityanalysis(eg,limitingtostudiesatlowestriskofbiasinasecondaryanalysis) 20 54
Narrative(discussionwithintext) 14 38
Exclusionofstudiesathighriskofbiasfrommainreviewsynthesis 5 14
Otherapproach 4 1120
Reference:Katikireddi SVetal.JEpidemiol CommunityHealth.2015Feb;69(2):189‐95. Availableathttp://jech.bmj.com/content/69/2/189
Table 4 Methods for Incorporating risk of bias assessments into reviews during synthesis
MetaAnalysis0 Nonewdatabutcangeneratenewhypothesis0 Validityofmeta‐analysisresultsreliedonthequalityofthedata
IntroductiontoMeta‐analysis0 Manyreadilyavailable(anduserfriendly)softwaretoallowpeoplewithoutmuchstatisticalbackgroundtoperformmeta‐analysis0 But..Mostdevelopedtocombinedrandomizedtrialresults
0 Doingameta‐analysisis(thus)veryeasybutdoingonewellishard0 Manypre‐analysissteps(e.g.,extractingquantitativedatafromtheliterature)requiresomebiostatisticsknowledge
0 Manymissingdata(intheliterature)mayberequiredtobeimputed/calculated
0 Interpretationsof(study‐level)meta‐analysisresultsaredifferentfromanalysisofpeople‐leveldata
22
0 Isitreasonable?0 Arethecharacteristicsandeffectsofstudiessufficientlysimilartoestimateanaverageeffect?
0 Typesofheterogeneity:0 Clinicaldiversity:Arethestudiesofsimilartreatments,populations,settings,design,etcetera,suchthatanaverageeffectwouldbeclinicallymeaningful?
0 Methodologicaldiversity:Arethestudiesofsimilardesignandconductsuchthatanaverageeffectwouldbeclinicallymeaningful?
0 Statisticalheterogeneity:Istheobservedvariabilityofeffectsgreaterthanthatexpectedbychancealone?
Heterogeneity(Diversity):toPoolornottoPool?
23
HeterogeneityIssuesaretheMainChallengesinConductingSRandMAofComplexIntervention
24Source:https://effectivehealthcare.ahrq.gov/topics/interventions‐tools‐guidance/abstract/
HETEROGENEOUS TREATMENT EFFECTS
IGNORE INCORPORATEESTIMATE(insensitive) EXPLAIN
FIXED EFFECTS
MODEL
DO NOT COMBINE WHEN
HETEROGENEITY IS PRESENT
RANDOM EFFECTS MODEL
SUBGROUP ANALYSES
META-REGRESSION(control rate, covariates)
DealingWithHeterogeneity:Meta‐analyticalTechniques