systematic reviews: the process, quantitative, qualitative and mixed methods reviews. edoardo...
DESCRIPTION
Health Libraries Australia Professional Development Day 2012 [Keynote address - part 2]TRANSCRIPT
www.joannabriggs.edu.au
Systematic Reviews: the process, quantitative, qualitative and mixed
methods reviews
Edoardo Aromataris
www.joannabriggs.edu.au
Systematic reviews• Different strokes...
– Where I’m coming from• Introduction to reviews and systematic reviews
– Terminology• Systematic steps• Journey beyond where you may be involved...• Where you’ll find differences in quantitative and
qualitative
www.joannabriggs.edu.au
Some answers!?• What is a systematic review?
www.joannabriggs.edu.au
Some answers!?• An attempt to identify, appraise and synthesise all
the empirical evidence that meets pre-specified eligibility criteria to answer a given research question.
• An attempt to sum up the best available research on a specific question. This is done by synthesising the results of several studies.
www.joannabriggs.edu.au
• “It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, updated periodically, of all relevant randomized controlled trials” Archie Cochrane, 1972
The Nature of Evidence
www.joannabriggs.edu.au
Cochrane Collaboration• Established 1993• Leaders in EBM, some 28,000 members• Focus on effectiveness of interventions and
therapies in medical practice– RCTs focus– More pluralistic – qualitative methods
• Organised into Cochrane Review Groups (CRG)• Cochrane Library
www.joannabriggs.edu.au
Campbell Collaboration (C2)• Focus on systematic reviews of effectiveness in
education, crime and justice, and social welfare• Coordinating groups (6)• Campbell Systematic Reviews
• Synthetic Reviews
www.joannabriggs.edu.au
• “It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, updated periodically, of all relevant randomized controlled trials” Archie Cochrane, 1972.
• Shift toward pluralistic, inclusive definitions of what evidence is, and subsequently of what evidence based practice is...
The Nature of Evidence
www.joannabriggs.edu.au
Joanna Briggs Institute• Established 1996• Feasibility, Appropriateness, Meaningfulness and
Effectiveness• Qualitative methods• JBC – 70+ Centres conduct reviews• JBI Library of Systematic Reviews• Leaders in EBHC and evidence implementation
www.joannabriggs.edu.au
Introduction to Systematic Reviews• Variations in conduct• Different questions, different methods of synthesis
– E.g. effectiveness, aetiology, harms, meta-ethnographic reviews, realist syntheses
• Not all are validated– Term used loosely
• Broad coverage, applicable to most reviews– Systematic!
www.joannabriggs.edu.au
Terminology• Literature review summarises, critiques, and
synthesizes articles while not using systematic methodology
• Systematic reviews adhere to explicit and rigorous methods to identify, critically appraise, and synthesise relevant primary/original studies.
(Krainovich-Miller, 2006:87)
www.joannabriggs.edu.au
Systematic Review• Also called “Research Synthesis”• Is an attempt to integrate empirical data for the
purpose of:– uncovering the international evidence and– producing statements about that evidence to guide
decision making• Requires explicit and exhaustive reporting of the
methods used in synthesis
www.joannabriggs.edu.au
Systematic Review
• ‘…an attempt to minimize the element of arbitrariness…by making explicit the review process, so that, in principle, another reviewer with access to the same resources could undertake the review and reach broadly the same conclusions’
(Dixon-Wood et al. 1997:157 quoted by Seers, 2005:102)
www.joannabriggs.edu.au
Bias
• ‘systematic deviation of results or inferences from truth’
• ‘an error in the conception and design of a study-or in the collection, analysis, interpretation, reporting, publication, or review of data-leading to results or conclusions that are systematically (as opposed to randomly) different from truth’ (Porta, 2008:18)
www.joannabriggs.edu.au
Meta-analysis • Quantitative evidence
– Questions of Effectiveness, Feasibility and/or Appropriateness
• Use of statistical methods to combine the results of various independent, similar studies
• More precise calculation of one estimate of treatment effect than could be achieved by any of the individual, contributing studies
• Only forms a part of the systematic review in which it appears
www.joannabriggs.edu.au
Meta-synthesis
• Qualitative evidence– Questions of Meaningfulness, Feasibility and/or Appropriateness
• Qualitative analysis of a number of independent qualitative research studies and text
• Use of qualitative methods of combining the findings of individual studies
• Only forms a part of the systematic review in which it appears
www.joannabriggs.edu.au
Qualitative Research Findings as Evidence for Practice• Qualitative evidence is of increasing importance in health
services policy, planning and delivery.• It can play a significant role in:
– understanding how individuals / communities perceive health, manage their own health and make decisions related to health service usage;
– increasing our understandings of the culture of communities and of health units;
– Informing planners and providers;– evaluating components and activities of health services that
cannot be measured in quantitative outcomes.
www.joannabriggs.edu.au
Systematic Review• The notion of and methods for establishing credibility in
systematic reviews has been extensively developed and debated
• In terms of quantitative evidence:– Emphasis on reducing bias and increasing validity– Degree of credibility established through critique and by applying
levels of evidence• In terms of qualitative evidence:
– Emphasis on rigour of research design and transferability– Degree of credibility established through critique and by applying
levels of credibility
www.joannabriggs.edu.au
Comprehensive/Mixed method Review• Series of questions• Combines both quantitative and qualitative findings
and addresses multiple forms of evidence• 2 or more types of evidence• Different approaches - particularly to integration of
qualitative findings
www.joannabriggs.edu.au
Characteristics of a SR• Protocol driven process• Clearly stated set of objectives with pre-defined eligibility
criteria for studies• Explicit, reproducible methodology • Systematic search that attempts to identify all studies that
would meet the eligibility criteria• Assessment of the validity of the findings of the included
studies • Systematic presentation, and synthesis, of the characteristics
and findings of the included studies(Green et al., 2008:6)
www.joannabriggs.edu.au
Steps in a Systematic Review• Formulate review question• Define inclusion and exclusion criteria• Locate studies• Select studies• Assess study quality• Extract data• Analysis/summary and synthesis of relevant studies• Present results• Interpret results/determining the applicability of results
(Egger & Davey Smith, 2001:25; Glasziou et al., 2004:2)
www.joannabriggs.edu.au
Question Development
• Reviews of effects & economics:– Population– Intervention/Exposure– Comparator– Outcome– Types of Study Design
• Reviews of qualitative & Textual data:– Population– Phenomena of Interest– Context
Divide question following the PICO/PICo model
www.joannabriggs.edu.au
Question?• Very important - guides entire review• Effectiveness? Meaningfulness?• Other components - How? Why? When?
www.joannabriggs.edu.au
Where to Start?• Conduct a search for previous Systematic
Reviews on your topic: has this review already been conducted or is a review already in progress? – Yes = stop and refine your question– No = proceed with your protocol
development
www.joannabriggs.edu.au
Protocol• Detailed review methods a priori• Guide process
– Reasoned approach to the question asked• Decrease biased post hoc changes
– Important to avoid “fishing”• Review proper may deviate
– Needs clear explanation how and why
www.joannabriggs.edu.au
Protocol Development• The usefulness and success of the review stems
from the robustness of the protocol• The protocol:
– Guides the specific direction of the review– Describes inclusion criteria– Identifies the appropriate search sources and
resources– Methods of appraisal, extraction and synthesis
www.joannabriggs.edu.au
Background• Describe the issue under review, including:
– Target population, interventions, outcomes, • Should concisely overview the main elements of the
review, and issues within the topic of choice• Provide adequate detail to justify the conduct of the
review and choice of inclusion criteria• Provide necessary definitions of important terms
and concepts
www.joannabriggs.edu.au
Inclusion Criteria: Quantitative data
• Study designs • Population characteristics• Intervention or exposure• Comparator - active or passive• Outcomes of interest
www.joannabriggs.edu.au
Inclusion Criteria: Qualitative & Text
• Study designs • Population characteristics• Phenomena of Interest• Context
www.joannabriggs.edu.au
Search Strategy
• Know and understand the question!• What is the relevant PICO / PICo?• Qualitative, quantitative or economic?• Feasibility, Appropriateness, Meaningfulness or
Effectiveness?• Other questions? Extra concepts? Less
concepts?
www.joannabriggs.edu.au
Search Strategy Steps• Initial Search
– initial search of MEDLINE, CINAHL, followed by analysis of text words in the title and abstract
• Second Search– all identified key words and index terms across all
databases• Third Search
– references of identified studies and unpublished studies
www.joannabriggs.edu.au
Search Strategy• Features of search strategy
– Sensitivity – ability to identify all the relevant studies– Specificity – ability to exclude irrelevant studies
• Inverse relationship between sensitivity and specificity– High sensitivity will tend to have low specificity– A large number of articles retrieved may not be relevant
to the review question
www.joannabriggs.edu.au
Searching• Depend on methods (to minimise bias, or not?)
– Exhaustive or Saturation based?– Limited or comprehensive?
• Impact on transparency and auditability• Where appropriate?
– How much is enough?– Are appropriate sources being covered?
www.joannabriggs.edu.au
Types of Sources
• What to use?– Scientific databases – Scientific Journals– Organisations– Websites– Libraries– Experts
www.joannabriggs.edu.au
Types of Resources• Peer reviewed journal articles
– Research– Opinion/commentary/letters
• Grey Literature– Google Scholar, Mednar
• Theses/Dissertations– DATAD
• Data – Statistics• Circulars• Reports
www.joannabriggs.edu.au
Databases and sources…• Medline/PubMed• Scopus/Embase• POPLINE• CINAHL• ERIC• PsychINFO
Consult your Librarian!
• ScienceDirect• TRIP• Wiley InterScience• SPORTDiscus• ISI Web of Science• + many more…!
www.joannabriggs.edu.au
Research and Trials Registers• Cochrane library
– CCTR• Clinical trials.gov• Controlled Clinical Trials• NHS Research Register• REGARD (database of the ESRC)
www.joannabriggs.edu.au
Grey literature
• Mednar• PsycEXTRA • OAIster • SIGLE• Google Scholar
www.joannabriggs.edu.au
Theses & Dissertations• Index to Thesis • Networked Digital Library of Theses and
Dissertations (NDLTD)• ProQuest Dissertations and Theses Database• EThOS - Beta - Electronic Thesis Online System• TROVE (ADTP)• +…
www.joannabriggs.edu.au
Government Websites• Are there government agencies that may have
evidence relevant to your review question?– NCCAM– CDC– NH&MRC– AHRQ– AIHW– +…
Search Logistics!
• Apply Search Strategy to databases
• Export to bibliographic software– E.g. Endnote
• Document process
Search Strategy
Search Sources
Export citationsto bibliographic software
www.joannabriggs.edu.au
Study Selection• Study Selection is an initial assessment that occurs
following the review Search• It addresses the question “should the paper be
retrieved?”– 2nd assessment that occurs after retrieval and addresses the
question “should the study be included in the review?” - this is CRITICAL APPRAISAL!
• It is essential to use two assessors in both the selection and critical appraisal processes to limit the risk of error
www.joannabriggs.edu.au
Selection Process• Aims to select only those studies that address the
review question and that match the inclusion criteria documented in your protocol
• Scan titles and abstracts• Err on the side of caution - Inclusive!• If uncertain? - Retrieve - scan full text• The selection should be:
– Transparent– Reproducible
www.joannabriggs.edu.au
Critical Appraisal• Aim is to establish validity
– To establish the risk of bias– Internal validity
• Every review must set out to use an explicit appraisal process. Essentially,– A good understanding of research design is required in
appraisers; and– The use of an agreed checklist or scale is usual.
Why Critically Appraise?
• Combining results of poor quality research may lead to biased or misleading results and understandings
1004 references
832 referencesScanned Ti/Ab
172 duplicates
117 studiesretrieved
715 do not meetIncl. criteria
82 do not meetIncl. criteria
35 studies forCritical Appraisal
www.joannabriggs.edu.au
Assessing the Risk of Bias• Numerous tools are available for assessing methodological
quality of clinical trials and observational studies.• Generally requires the use of a specific tool for assessing risk
of bias in each included study. • ‘High quality’ research methods can still leave a study at
important risk of bias (e.g. when blinding is impossible)• Some markers of quality are unlikely to have direct
implications for risk of bias (e.g ethical approval, sample size calculation)
Type of bias Quality assessment
Population
Allocation
Selection Allocation concealment
Treatment Control
Performance Blinding Exposed to intervention
Not exposed
Detection Blinding Population Population
Attrition ITT follow up Follow up Follow up
JBI-MAStARI Instrument
www.joannabriggs.edu.au
Critical Appraisal• Exclude studies on basis of appraisal?• Explore impact through sensitivity
analyses/subgroups
www.joannabriggs.edu.au
Critical Appraisal & Qualitative Review• General acceptance of need for quality• Ongoing debate around the role of appraisal• Particular focus of debate on role of scales and sum
scores in appraisal• In practice, appraisal instruments for qualitative
research tend to focus on establishing the degree to which the evidence applies to practice (transferability) rather than internal validity (credibility)
www.joannabriggs.edu.au
Considerations in Data Extraction
• Source - citation and contact details• Eligibility - confirm eligibility for review• Methods - study design, concerns about bias• Participants - total number, setting, diagnostic criteria • Interventions - total number of intervention groups• Outcomes - outcomes and time points• Results - for each outcome of interest: sample size, etc• Miscellaneous - funding source, etc
www.joannabriggs.edu.au
Data Extraction• The data extracted for a systematic review are the
results from individual studies specifically related to the review question.
• Difficulties related to the extraction of data include:– different populations used– different outcome measures– different scales or measures used– interventions administered differently– reliability of data extraction (i.e: between reviewers)
www.joannabriggs.edu.au
Extracting Findings for Qualitative reviews• The units of extraction in this process are specific
findings and illustrations from the text that demonstrate the origins of the findings
• A finding is defined as: A conclusion reached by the researcher(s) and often presented as themes or metaphors
www.joannabriggs.edu.au
Analyses• Depend on data collected/question• Maybe simply narrative• Meta analysis?• Qualitative syntheses?• Mixed methods
www.joannabriggs.edu.au
Analyses and reporting– What interventions/activities have been evaluated– The effectiveness /appropriateness /feasibility
/meaningfulness of the intervention/activity– Contradictory findings and conflicts– Limitations of study methods– Issues related to study quality– The use of inappropriate definitions– Specific populations excluded from studies– Future research needs
Meta Analysis &Meta Synthesis
1004 references
832 referencesScanned Ti/Ab
172 duplicates
117 studiesretrieved
715 do not meetIncl. criteria
82 do not meetIncl. criteria
35 studies forCritical Appraisal
26 studies incl.in review
6 studies incl.in meta analysis
20 studies incl.in narrative
9 excludedstudies
www.joannabriggs.edu.au
Statistical methods for meta-analysis• Quantitative method of combining results of
independent studies• Aim is to increase precision of overall estimate• Investigate reasons for differences in risk estimates
between studies• Discover patterns of risk amongst studies
www.joannabriggs.edu.au
When is meta-analysis useful?• If studies report different treatment effects.• If studies are too small (insufficient power) to detect
meaningful effects.• Single studies rarely, if ever, provide definitive
conclusions regarding the effectiveness of an intervention.
www.joannabriggs.edu.au
When meta-analysis can be used• Meta analysis can be used if studies:
– have the same population– use the same intervention administered in the same way.– measure the same outcomes
• Homogeneity– studies are sufficiently similar to estimate an average
effect.
Taken from Egger, M. et al. BMJ 1998;316:140-144
Example meta-analysis
www.joannabriggs.edu.au
• Analysis and synthesis of qualitative studies • Based on processed data• Aim of meta-synthesis
• is to assemble findings; • categorise these findings into groups on the basis of
similarity in meaning; • aggregate these to generate a set of statements that
adequately represent that aggregation.
Meta-Synthesis
www.joannabriggs.edu.au
Data Synthesis Involves:• Step 1: Identifying findings• Step 2: Grouping findings into categories; and• Step 3: Grouping categories into synthesised findings
STEP 2: CATEGORIES
STEP 3: SYNTHESISED FINDINGS
FIRST ORDER ANALYSIS
SECOND ORDER INTERPRETATION
THIRD ORDER INTERPRETATION
QARI METAGGREGATION
META ETHNOGRAPHY
Comprehensive/Mixed method Review
FOCUS technical brief #25, Angela Harden, 2010
www.joannabriggs.edu.au
Interpretation• Quality, strength and applicability of evidence for
outcomes• Applicability for stakeholders/policy makers• Qualitative evidence can aid in interpretation• How, why, barriers, facilitators, experience• Conclusions/recommendations should be drawn
based on available evidence• May be none available!
The JBI Software
System for theUnified
Management of theAssessment and
Review ofInformation
Consists of the following components
Comprehensive Review and Management System
Keeps all review information together and generates a
report that may contain elements from all other
SUMARI modules
• Protocol• Reviewers• Search strategy• Bibliography - retrieved studies• Bibliography - non selected
studies
Qualitative Assessment and Review Instrument
Meta-Analysis of Statistics Assessment and Review Instrument(Used in place of Review Manager for non-Cochrane reviews)
Narrative Opinion and Text Assessment and Review Instrument
Analysis of Cost Technology and Utilisation Assessment and Review Instrument