grading the quality of a body of evidence for complex ... · prof paul montgomery university of...

Department or office title: change on Slide Master

The image part with relationship ID rId3 was not found in the file.

Grading the quality of a body of evidence for

complex interventions

EUSPR Berlin 2016

Prof Paul MontgomeryUniversity of Oxford



Outline Complexity concepts in intervention research

Implications of complexity for systematic reviews

Applying the GRADE approach to complex interventions: challenges and user-proposed adaptations

Project on the GRADE extension for complex interventions

Your thoughts and feedback



Complexity concepts in intervention research



Background (1)UK Medical Research Council Guidance (2000)

Continuum of Increasing EvidenceContinuum of Increasing Evidence




Feasibility & Piloting1. Testing procedures2. Estimating recruitment & retention3. Determining sample size

Development Evaluation

Implementation

1. Identifying evidence base2. Identifying or developing theory3. Modelling process and outcomes

1. Assessing effectiveness2. Understanding change process3. Assessing cost effectiveness

1. Dissemination2. Surveillance & monitoring3. Long-term follow up




ContextContextual factors which shape theories of how the intervention worksContextual factors which affect (or may be affected by) implementation, mechanisms & outcomesCausal mechanisms present within the context which act to sustain the status quo/potential effects

Descriptionof interventionand its causalassumptions

OutcomesImplementation process:How delivery is achieved?

What is delivered:Fidelity, dose, adaptations,

reach

Mechanisms of impact:- participant responses - Mediators- Unanticipated pathways and consequences



Perspective 1: Interventions (Medical Research Council: 2008)

• Number of and interactions between components within the experimental and control interventions

• Number and difficulty of behaviours required by those delivering or receiving the interventions

• Number of groups or organisational levels targeted by the intervention

• Number and variability of outcomes

• Degree of flexibility or tailoring of the intervention permitted

Craig P. et al. (2008). Developing and evaluating complex interventions. New guidance.Medical Research Council



Perspective 2: Causal Pathways(J Clin Epidemiol special issue: 66; 2013)

• Complexity in participant responses – when the intervention effects are expected to vary because of the different characteristics of the participants

• Complexity in intervention – when the intervention effects are expected to vary because of the different characteristics and elements of the intervention itself

• Complexity in implementation – when the intervention effects are expected to vary because of the implementation processes

• Complexity in context – when the intervention effects are expected to vary because of the different characteristics of the settings or contexts in which they are implemented



Perspective 3: Interventions in complex adaptive systems (slide taken from Dr Eva Rehfuess)

CONTEXT• Geographical• Epidemiological• Socio-cultural• Socio-economic• Ethical• Legal• Political

INTERVENTION• Theory• Components• Execution

COMPARISON• Business as usual

IMPLEMENTATION• Organisation/structure• Financing• Policy

OUTCOME• Health• Non-health

POPULATION• Biological, social• Organisational

SYSTEM PROCESSES• Multiple interactions• Feedback loops• Phase changes• Emergent properties



Implications of complexity for systematic reviews



Review question formulation

Interventions within systems

• what works vs. what happens?

• what works, for whom, under what circumstances?

• it worked there vs. will it here?

“If evidence-based policy is to do its job then, it is best to construe evidence widely enough to cover all the facts without which you will not have a good argument”

Nancy Cartwright

Can/should PICO be extended and/or elaborated to incorporate the complex system perspective?



Evidence synthesis methods


• Quantitative synthesis methods

• Qualitative synthesis methods

• Mixed-method synthesis

“If we are to achieve things never before accomplished, we must employ methods never before attempted”

Francis Bacon



Using a combination of evidence synthesis methods

(J Clin Epidemiol series)

Quantitative synthesis to determine effects,

explain/explore context

Qualitative synthesis to configure/summarise

integrate data

Qualitative synthesis to develop explanatory

models or theory

Mixed-method synthesis to determine effects,

explain/explore context

Meta-analysis, meta-regression or narrative

summary

ProductPooled effect size and/or

description of single studies

ProductIntegrated synthesis of

quantitative & qualitative evidence

Thematic analysis without theory generation

e.g. framework synthesis

ProductAggregated/configured narrative findings from

source papers

Thematic analysis with theory generation Meta-ethnography

ProductExplanatory theory,

interpretive framework/ mechanism

Realist reviewEPPI approach

Narrative synthesis

ProductIntegrated synthesis of

quantitative & qualitative evidence

Petticrew et al. Synthesizing evidence on complex interventions: how meta-analytical, qualitative and mixed-

method approaches can conrtibute. J Clin Epidemiol. 2013:66;1230-1243

Quantitative synthesis Qualitative synthesis Qualitative synthesis Mixed-method synthesis



Grading the quality of a body of evidence


• Hierarchy vs. appropriateness

• Internal validity vs. external validity

• Confidence that it worked there vs. confidence that it will work here

“…the rules are actually good at identifying policies that work, that is policies that work somewhere; the failing is rather that it is not designed to deliver the key facts required to conclude that it will work here”

Nancy Cartwright



Applying the GRADE approach to complex interventions



The GRADE Methodology and ProcessQuestion formulation (PICO)

Search & retrieval of relevant studies

Evidence synthesis

Rating the overall quality of evidence

Rating the quality of evidence for each outcome

RCTs – high, Observational – Low

Grading recommendations for practice - Problem priority - Acceptability- Benefits/harms - Preferences/values - Quality of evidence - Resource use- Feasibility - Equity

Downgrading Upgrading- Risk of bias - Large effect- Inconsistency - Dose-response- Imprecision - All plausible - Indirectness residual- Publication bias confounding

Quality of evidence is defined as the extent of our confidence that the estimates of the effect

are correct

Comparative effectiveness

Meta-analysis of RCTs; narrative summary

11

22

44

66 DECIDE

Evidence synthesis33

Rating the overall quality of evidence55



Definitions of the GRADE quality of evidence ratingsLevel GRADE definition AHRQ/EPCs

High We are very confident that the true effect lies close to that of the estimate of the effect

We are very confident that the estimate of effect lies close to the true effect for this outcome

Moderate We are moderately confident in the effect estimate: the true effect is likely to be close to the estimate of effect, but there is a possibility that it is substantially different

We are moderately confident that the estimate of effect lies close to the true effect for this outcome. The body of evidence has some deficiencies. We believe that the findings are likely to be stable, but some doubt remains.

Low Our confidence in the effect estimate is limited: the true effect may be substantially different from the estimate of the effect

We have limited confidence that the estimate of effect lies close to the true effect for this outcome. The body of evidence has major or numerous deficiencies (or both). We believe that additional evidence is needed before concluding either that the findings are stable or that the estimate of effect is close to the true effect.

Very Low We have very little confidence in the effect estimate: the true effect is likely to be substantially different from the estimate of effect

Insufficient: we have no evidence, we are unable to estimate an effect, or we have no confidence in the estimate of effect for this outcome. No evidence is available or the body of evidence has unacceptable deficiencies, precluding reaching a conclusion.

Guyatt et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011(64);12:1283-1293Berkman et al. Grading the strength of a body of evidence when assessing health care interventions: an EPC update. J Clin Epidemiol 2016 (68):1312-1324



Challenges of using GRADE in social interventions

• Inappropriate use of terminology Example: use of terms, such as patients and clinicians

• Irrelevant definition and meaning of quality/confidenceConcern: the effects are critically influenced by modes of delivery and contextual factorsAlternative definition: “confidence that the effect is meaningful across a range of plausible implementation contexts”

• Inappropriate interpretations of the levels of evidence qualityConcern: misinterpretations of “low quality evidence” by policymakers?

1. GRADE terminology and definitions

Rehfuess & Akl (2013)




• Scarcity of RCTs to address effectiveness questionsConcern: GRADE is inflexible when RCTs are not feasible (rigour versus feasibility)

• Non-randomised studies versus other observational studiesConcern: GRADE doesn’t differentiate between designs less prone to bias (e.g. ITS)and other observational studiesAlternative: the selected designs enter the assessment as “moderate”

• Selection of an appropriate body of evidenceConcern: how to prioritise between one large RCT and many Non-RCTs conducted in different contexts? Implications for generalisability?

2. Evidence base and rigour hierarchy

Harder et al. (2015); Movsisyan et al. (2015); Rehfuess & Akl (2013)




• Interpretation of InconsistencyConcern: how to interpret heterogeneity for multi-component interventions when eitherlumping or splitting?

• Judgment of IndirectnessConcern (1): how to judge about the degree of indirectness for multi-component interventions when either lumping or splitting?Concern (2): how to prioritise between many outcomes (short-term versus long-term) and outcome measures, and what are the implications of this for indirectness?

• Risk of bias assessment Concern (1): downgrading evidence for lack of blinding, when impossible to blind(rigour versus feasibility)Concern (2): study designs used for these interventions do not have risk of bias tools for consistent use, which complicates the GRADE assessment (e.g. NRS, SSED, etc.)

3. Specific criteria

Harder et al. (2015); McPheeters et al. (2015); Movsisyan et al. (2015); Rehfuess & Akl (2013)




• Use of non-epidemiological evidenceConcern (1): how to incorporate evidence on implementation & context to facilitate context-specific effectiveness assessment in GRADE?Alternative (1): using non-epidemiological evidence not as a separate low quality evidence, but to augment the credibility of epidemiological evidence(e.g. a causal-chain approach, applicability assessment)

• Insufficient possibilities for upgrading observational evidenceAlternative (1): upgrade for consistency across study designs, settings, research groupsAlternative (2): upgrade for analogy from “parallel evidence”, such as evidence from related population groups, interventions

4. Making the best use of available evidence

Harder et al. (2015); Movsisyan et al. (2015); Rehfuess & Akl (2013)



The GRADE Extension for CI?Question formulation (PICO)

Search & retrieval of relevant studies

Evidence synthesis

Rating the overall quality of evidence

Rating the quality of evidence for each outcome

RCTs – high, Observational – Low

Grading recommendations for practice - Problem priority - Acceptability- Benefits/harms - Preferences/values - Quality of evidence - Resource use- Feasibility - Equity

Downgrading Upgrading- Risk of bias - Large effect- Inconsistency - Dose-response- Imprecision - All plausible - Indirectness residual- Publication bias confounding

Quality of evidence is defined as the extent of our confidence that the estimates of the effect

are correct

What happens?

A spectrum of evidence synthesis methods?

11

22

44

66 DECIDE

Evidence synthesis33

Rating the overall quality of evidence55



Project on the GRADE extension for complex interventions



Project Aim and Objectives

Revise aspects of the GRADE methodology to enable the best use of available evidence to inform decision-making on the effects of complex social interventions

- revise GRADE terminology & definitions

- reconsider the evidence hierarchy within GRADE

- rethink the criteria for rating the quality of evidence



Project Protocol

PHASE 1Project Launch

- Finalise the team- Build collaboration- Systematic review

Start Date: 01.01. 2016 End Date: 30.06.2018


STEERING & COORDINATION

DISSEMINATION

PHASE 2Online Expert Panel

- Identify participants- Conduct the panel- Data analysis

PHASE 3Consensus Meeting

- Pre-meeting - Host the meeting- Data analysis

PHASE 4Write-up & Testing

- Draft documents- Feedback & revise- Finalise documents



Project Executive• Dr Erik von Elm –IUMSP, Lausanne, Switzerland

• Dr Eva Rehfuess – Institute of Medical Informatics, Munich, Germany

• Prof Geraldine Macdonald – University of Bristol, Bristol, UK

• Dr Jane Dennis – Research Synthesis Ltd, Bristol, UK

• Prof Paul Montgomery – University of Oxford

• Dr Sean Grant – RAND Corporation, Santa Monica, USA

• Dr Susan Norris – Guideline Review Committee Secretariat, WHO



PHASE 1. Systematic Review

Objective: to investigate the content and development of the systems for grading the quality of a body of evidence (i.e. an accumulation of primary research evidence that addresses a common scientific issue) on the intervention effectiveness

- Published documents introducing a systematic procedure and explicit standards for assessing the quality of the body of evidence for statements underpinning the effectiveness of interventions



PHASE 1. Systematic ReviewSearch Strategy

1. Systematic searches in scientific databases (1995 to 2016)• Applied Social Sciences Index (ASSIA)• EMBASE (Ovid)• Cochrane Methodology Register (Cochrane Library)• MEDLINE (Ovid)• PsycINFO (Ovid)• Scopus Social Sciences• Social Sciences Citation Index (Web of Knowledge)• SCIE Social Care Online

2. Searches in 44 websites of key stakeholder organisations3. Consulting experts to ensure comprehensiveness of the search4. Searches of the bibliographies of all the eligible publications



Project Protocol

PHASE 1Project Launch

- Finalise the team- Build collaboration- Systematic review



STEERING & COORDINATION

DISSEMINATION

PHASE 2Online Expert Panel

- Identify participants- Conduct the panel- Data analysis

PHASE 3Consensus Meeting

- Pre-meeting - Host the meeting- Data analysis

PHASE 4Write-up & Testing

- Draft documents- Feedback & revise- Finalise documents



Questions?Feedback



Additional slides



Target Audience for the GRADE Extension

1. Producers of systematic reviews of complex interventions- knowledge base for decision-makers(e.g. Cochrane and Campbell Reviews, EPPI-Centre Reviews)- evidence synthesis as part of guideline development (e.g. WHO, NICE Collaborating Centre for Social Care)

2. Systematic review methodologists(e.g. Cochrane Collaboration, EPPI-Centre)

3. Those commissioning or using systematic reviews of complex interventions to inform practice & policy- governmental (e.g. DFID)- non-governmental (e.g. 3ie)



What Should Inform the GRADE Extension?1. Standards that are already established and in use (systematic review

of relevant evidence grading systems)

2. Literature on the challenges and user-proposed modifications for the GRADE extension?

3. Recent publications and projects focusing on methodological developments in systematic reviews of effectiveness of (complex) social interventions:JCE series, MICCI, INTEGRATE-HTA, WHO,

4. International online expert panel

5. Consensus meeting

6. Effectiveness review of a complex social intervention (case study and also for future testing of the extension – WHO?)



Social InterventionsInterventions

Services, policies, multi-stage programmes

MediatorsPsycho-social processesagents within social systems

OutcomesHealth, functioning, well-

being

1

2

3

Delivered to individualsExample: Cognitive-behavioural therapy

Delivered to populations, communitiesExample: Community-based interventions

The intervention operates by alteringpsycho-social processes: - cognitions, emotions, behaviours- interpersonal relationships, norms, attitudes, values- social & physical aspects of environments

Health: physical & mental

Social: Education, work, social relationships, community involvement, poverty,

Well-being: quality of life, spirituality, life, socio-economic, satisfaction, self-determination

Adapted from: Grant et al. (2014). Development of CONSORT-SPI.England et al. (2015). Psycho-social interventions for mental health and substance use disorders. IOM (Institute of Medicine).



Practice domains which investigate social interventions

Intervention Psycho-social processes Outcomes

Public Health Built environment interventions

Structural, psycho-social, e.g. self-efficacy, social support, motivation

Physical activity, mental health, quality of life

Education Technical andvocational education

and training

Motivation, knowledge skills, behaviours

Employability and employment of young people

Crime &Justice

Restorative justiceconferencing

Emotions and behaviours Repeat offending, victimsatisfaction

Psychology Parenting interventions Cognitions, relationships, emotions, behaviours

Conduct behaviour, parent and child mental health,

wellbeing

Social Work & Welfare

Kinship care Social environment, positive bonds with the caregiver

Mental health, educational attainment

International Development

Saving promotion interventions in LMICs

social environment, norms, attitudes, self-efficacy, self-control, behaviours, e.g. budgeting

Poverty, Educational attainment, health

grading the quality of a body of evidence for complex ... · prof paul montgomery university of...

Documents