medicine, nursing and health sciences methods for synthesizing evidence of the effects of healthcare...

Medicine, Nursing and Health Sciences

Methods for synthesizing evidence of the effects of healthcare interventions in systematic reviews of complex interventions (including statistical approaches)Joanne McKenzie, School of Public Health and Preventive Medicine19th Cochrane ColloquiumMadrid 2011

October 2011Methods for synthesizing evidence of the effects of healthcare interventions 2

Google’s take on complexity …

www.nicholsoncartoons.com.au

Acknowledgements

Sue Brennan, Monash University, Australia

Sophie Hill, La Trobe University, Australia


Outline

Components of complexity in a systematic review

Summary to synthesis (pros and cons)

– Available methods

– Outcome categorisation

Conclusions and questions raised


Splitting Lumping

ConditionOne condition (e.g. diabetes)

Any condition

One setting (e.g. primary care)

Setting

All settings

Narrow(e.g. audit cycles)

Intervention

Any form(e.g. any QI

intervention)

One design(e.g. RCT)

Study design

Multiple designs

(e.g. RCT, ITS, CBA)

Consistent outcomes (e.g. test ordering)

Diverse outcomes

Outcome


Summary

Vote counting Summary of effect estimates

Meta-analysis

- texttabular

Synthesis

- texttabularharvest plots

- descriptive statisticsbox and whisker plots

- meta-analysispredictive intervalsforest plots

- sub-group analysismeta-regressiongraphical approaches

Exploring heterogeneity


Summary

Results of studies summarised:

• in the text of a publication without the use of a synthesis method

• in tables, providing a structured method for presenting data

Pros

Text: Provides an assembly of the available research meeting the inclusion/exclusion criteria.

Tables: More likely to report all results of all outcomes (i.e. may be less likely to selectively include results).

Results available for others to synthesize.

Cons

Text: Results summarised, not synthesized.

Little structure to reporting results may lead to selective reporting (privileging of findings above others).

Interpretation of results difficult/not possible.

Tables: Results summarised, not synthesized.

Overwhelming amount of information which is difficult for a reader to interpret (often multiple outcomes per study).


Splitting Lumping

ConditionAny condition/aspect of care: preventive care, diabetes, asthma, hypertension, HIV, renal failure, palliative care, coronary heart failure, stroke, pain management, falls prevention, neonatal infection, anticipatory guidance, compensation, wait times

Setting

Any setting: ambulatory (22), inpatient/nursing home (10), mixed clinical (3), educational (4)

InterventionAny form: QI education for trainees (10), QI education for nontrainees (5), ‘other’ interventions with an educational component (5), education within a QI collaborative (19)

Study design

Multiple designs: RCT (8), nonrandomised trial (14), pre/post or time series (17)

Any outcome: attitude (6), knowledge (10), skill/behaviour (6), process (27), patient (18)

Outcome

Quality improvement education for clinicians

Boonyasai, JAMA 2007

[Boonyasai JAMA 2007]

Study 1

Study 2

Study 3

• 27/39 studies measured process outcomes (3 example studies)

• 15 clinical areas


Outcome categorisation (EPOC)

Example CQI review:

Healthcare professional performance (binary, continuous)

– e.g. adherence to recommended practice

Patient outcomes (binary, continuous)

– e.g. pain, quality of life, function, mortality

– e.g. patient experience of care, patient evaluation of care co-ordination, length of stay

Other outcomes

– e.g. resource use

[Brennan Cochrane Database Syst Rev 2009]


[Hill Wiley-Blackwell 2011]

Table 4.2: Outcomes of importance to consumers, communication and participation: a new taxonomy

Groups and orientation

Outcome category

(a) Consumer Knowledge and understandingCommunicationInvolvement in care Evaluation of careSupportSkills acquisitionHealth status and well beingHealth behaviourTreatment outcomes

(b) Healthcare provider

Knowledge and understandingConsultation processes

(c) Health service delivery

Service delivery levelRelated to researchSocietal or governmental

Outcome categorisation (CCRG)


SynthesisVote counting:• “Is there any evidence of an effect?”

• # studies showing harm compared with # studies showing benefit (regardless of stat. sig. or size of results)

• Sign test used to assess the stat. sig. of evidence of an effect in either direction

Pros

Provides a method for synthesizing effects when standard meta-analytical methods difficult to apply (e.g. variances of effect estimates not available).

Cons

Provides no information on the magnitude of effects (e.g. equal importance given to risk difference of 5% and 50%) .

No account of differential weighting across the studies.

Problems when stat. sig. used to define # positive and # negative studies (unit of analysis errors, underpowered studies).


[Boonyasai JAMA 2007]

Effectiveness of teaching quality improvement to clinicians

Harvest plots

Crowther Res Syn Meth 2011


SynthesisSummary of effect estimates:• “What is the range and distribution of effects?”

• ‘Median-of-medians’ approach (EPOC).

• One outcome chosen per outcome category (selection process independent of result & stat. sig.). Effect size associated with this outcome used to ‘characterise’ the outcome of the study.

Pros

Provides a method for synthesizing results when difficult to undertake a meta-analysis (e.g. missing variances of effects, unit of analysis errors).

Provides information on the magnitude and range of effects (IQR, range).

Cons

Does not weight effects; small studies are as influential as large studies.

Doesn’t use all available data for a particular outcome category.


[Farmer Cochrane Database Syst Rev 2008]

Printed educational materials: effects on professional practice and health care outcomes


[Jamtvedt Cochrane Database Syst Rev 2006 ]

Audit and feedback: effects on professional practice and health care outcomes


SynthesisMeta-analysis:• “What is the average intervention effect?” (random effects meta-analysis)

• Prediction intervals can be calculated to complement information of random effects meta-analysis; “What is the potential effect of an intervention in an individual study?”

Pros

Provides a combined estimate of average intervention effect (random effects), and certainty in this estimate (95% CI).

Weights estimates of effect; small studies are (generally) less influential compared with large studies.

Predictive intervals can be calculated; helpful when there is unexplained heterogeneity.

Forest plots display study effect estimates and CIs; can display pooled effect; familiar.

Cons

Requires variances of the effects.

Argued that a meta-analytic estimate (average effect) may be of little value when there is heterogeneity. Particularly if there is inconsistency in the direction of effect.


Exploring heterogeneitySub-group analysis, meta-regression, other statistical approaches:• “What factors modify the size of the intervention effect?”

• Can be used to investigate components (‘active ingredients’) of multifaceted interventions which may modify effects.

Pros

Provides hypotheses regarding what (set) of factors might be necessary for the intervention to be effective.

Cons

Observational analysis; may suffer confounding bias; aggregation bias; overfitting and spurious claims of association.

Investigation of intervention components when there are many is difficult (e.g. assumptions of additivity, correlation between combinations of components).

Requires variances of effects, measurement of factors.

Technical issues with baseline compliance.


[Jamtvedt Cochrane Database Syst Rev 2006]

Audit and feedback: effects on professional practice and health care outcomes

Re-analysis of audit & feedback review

[Gardner Soc Sci Med 2010]

Authors used theory (behaviour change) to categorise intervention components (feedback, performance target, action plan), and investigated if components modified the effects.


Conclusions

Diversity of interventions, settings, conditions, outcomes, and study designs complicates the synthesis of evidence.

A range of ‘synthesis’ approaches are available; some are clearly better than others.

Limitations in quantitative synthesis should be acknowledged, but may be preferable to “qualitative interpretation of results, or hidden quasi-quantitative analysis …” [Ioannidis BMJ 2008]

Before making a decision not to synthesize data, review authors should consider what readers/decision makers might do (e.g. selection of favourable effects, count up #favourable results or stat. sig. results).


Conclusions






Questions raised

Are arguments for not undertaking a meta-analysis based on too much clinical and methodological heterogeneity consistent with the ‘median-of-medians’ approach?

Are there other statistical approaches that may make better use of available data? E.g. meta-regression methods that adjust for correlated effects within studies (e.g. Hedges Res Syn Meth 2010).

The measures of effect used in complex reviews typically adjust for baseline imbalance (e.g. adj. RR, adj RD, adj OR). Do these estimators achieve the desired effect?

Will New Zealand win the 2011 Rugby World Cup?


References

Boonyasai et al. Effectiveness of teaching quality improvement to clinicians: a systematic review. JAMA 2007;298(9):1023-37.

Brennan et al. Continuous quality improvement: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2009, Issue 4. 10.1002/14651858.CD003319.pub2.

Crowther M, Avenell A, MacLennan G, Mowatt G. A further use for the Harvest plot: a novel method for the presentation of data synthesis. Res Syn Meth 2011

Gardner B et al. Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Soc Sci Med 2010;70(10):1618-25.

Farmer et al Printed educational materials: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2008(3): CD004398.

Hedges et al. Robust variance estimation in meta-regression with dependent effect size estimates. Res Syn Meth 2010;1(1):39-65.

Hill et al. Identifying outcomes of importance to consumers, communication and participation. In Hill S (ed). The Knowledgeable patient: Communication and participation in health. Wiley-Blackwell 2011.

Ioannidis JP et al. Reasons or excuses for avoiding meta-analysis in forest plots. BMJ 2008;336(7658):1413-5.

Jamtvedt et al. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2006(2):CD000259.

medicine, nursing and health sciences methods for synthesizing evidence of the effects of healthcare...

Documents

synthesizing evidence

cons text

australia slide

heterogeneity slide

synthesis method

presenting data pros

health sciences methods

synthesis pros