data distancing series: systematic reviews & meta-analyses ... · systematic reviews &...
TRANSCRIPT
Data Distancing Series: Systematic Reviews & Meta-Analyses - Part 1
Dr. Michal Perlman, Samantha Burns & Gabriella Nocita May 13, 2020
Outline – Day 11. Integration of Evidence: Why and How?
2. Overview of Types of Integrating of Evidence
3. Benefits of Systematic Reviews and Meta-Analyses
4. Being a Critical Consumer
5. Searching for Systematic Reviews and Meta-Analyses
Integration of Evidence
● Science requires replication of findings
● Need to see consistency across studies
● But sometimes it’s hard to tell what the pattern of results is
● Need to integrate across results from different studies
● There are different ways to do this that vary in how systematic and labour
intensive they are
● Focusing today on systematic review and meta-analysis
Types of Integrations of EvidenceNarrative Review Paper
● Qualitative summaries of evidence
● Usually written by experts
● Typically, involve informal, subjective methods of collecting and
interpreting studies
● Often do not explicitly describe how the reviewers searched or selected
their studies
● Often does not include quality assessments of the selected studies
● Tend to selectively cite literature that reinforces preconceived notions
Types of Integrations of EvidenceSystematic review
● Exhaustive search for primary studies on a focused topic
● Selection of studies using clear and reproducible eligibility criteria
● Critical appraisal of studies for quality
● Synthesis of results according to a pre-determined and explicit method
● Usually qualitative in nature
Types of Integrations of EvidenceMeta-analysis
● Usually the final step in a systematic review
● The statistical pooling of data across studies to generate summary
(pooled) estimates of effects
● The term ‘effect’ refers to any measure of association between exposure
and outcome (e.g. odds ratio)
Comparative OverviewSystematic Review Narrative Review
Question Focused on a single question Not necessarily focused on a single question, but may describe an overview
Protocol A peer review protocol or plan is included
No protocol is included
Background Both provide summaries of the available literature on a topicObjectives Clear objectives are identified Objectives may or may not be identified
Inclusion and exclusion criteria
Criteria stated before the review is conducted
Criteria not specified
Search strategy Comprehensive search conducted in a systematic way
Strategy not explicitly stated
Process of selecting articles Usually clear and explicit Not described in a narrative reviewProcess of evaluating articles
Comprehensive evaluation of study quality
Evaluation of study quality may or may not be included
Process of extracting relevant information
Usually clear and specific Not clear or explicit
Results and data synthesis Clear summaries of studies based on high quality evidence
Summary based on studies where the quality of the articles may not be expected. May also be influenced by the reviewer's theories, needs and beliefs
Discussion Written by an expert or group of experts with a detailed and well grounded knowledge of the issues
Process for Conducting Systematic Reviews/Meta-Analysis
What Makes Systematic Reviews and Meta-Analyses Great Research Tools?
Great Methodology For...● Systematic reviews and meta-analyses integrate findings across all
available research in a given area● This makes them great for:
○ Determining the extent to which a topic has been researched○ Quickly getting up to speed on a topic○ Finding key researchers and studies in a given area○ Understanding the pattern of results across studies ○ Identifying gaps in literature and future directions○ Coming up with your own research question○ Using as a starting point for your own literature review
● To reap these benefits, we have to be critical consumers
Important Considerations
1. Quality of Studies: Risk of Bias
2. Publication/Reporting Bias
3. Heterogeneity
4. Interpretation of Results
Assessment of Study Quality
● Very important step to determine how to be interpret the results
● Trying to avoid: “garbage in/garbage out”
● There are standardized tools for doing this○ Can use scores as inclusion criteria
○ Common Examples are the Newcastle-Ottawa and Cochrane Risk of Bias
○ A 2017 Systematic Review found 18 different risk of bias assessments
Ask yourself:
● Did the researchers assess study quality? If so, how?
● If not, of what quality do the included studies appear to be?
Publication/Reporting BiasesSignificant/Positive studies are more likely to be:
● Submitted for publication
● Accepted for publication
● Written in English
● Published more quickly
● Cited more often
Ask yourself:
● Did the authors account for publication or reporting bias? If so, how?
● What types of studies did they include? Do these studies introduce bias?○ If so, do they mention this as a limitation in the discussion?
Heterogeneity in Study Results
● Issues of heterogeneity in studies and results are paramount
● Combining heterogeneous effects (e.g. ,some strong positive associations
and some negative associations) can be very misleading
● If you combine apples and oranges you will end up with fruit salad. We
are social scientists and don’t want to make fruit salad…
Heterogeneity in Study Results● There are statistics that test for heterogeneity ● If it is too great researchers should not force results into a single meta-
analysis ● Instead they should try to find explanations for the observed
heterogeneity
Ask yourself:
● Did the authors make “fruit salad”?● Do the effects/outcomes the authors combined seem to make sense?● Were the authors too conservative in what they combined? Were they too
lax in what they combined?● Was their combination strategy different for the systematic review vs. the
meta-analysis?
Interpretation of Results● Discussing limitations in the existing literature and in current study is key
○ Ask yourself:■ Do the authors include limitations of literature? What about of their own study?■ If not, what are some such limitations (e.g., outdated studies, all cross-sectional)?
● Implications○ Ask yourself:
■ Do the authors discuss the implications of their integration of results?■ If so, do they make sense given what else is known in the field?■ If not, how might the results influence areas important to the field (e.g., public
policy, resource allocation, how variables are defined)
● Future directions for research○ Ask yourself:
■ Do the authors discuss where to go from the results?■ How can you use the results to inform gaps in literature and future studies?
Making Sure it’s Relevant: Where and How to Search for a Meta Analysis
How to Search for a Systematic Review/ Meta-Analysis
1. Cochrane Reviews: https://www.cochranelibrary.com/search
2. Campbell Systematic Reviews: https://onlinelibrary.wiley.com/journal/18911803
3. Library Databases with Meta-Analysis as a search term
4. Search the topic of interest in a database and filter by publication
type
Activity
Search for a Meta-Analysis in your field of interest
OR
Watch Demo of a Search
Coming Up Next…● How to conduct a systematic review and meta-analysis
○ Step-by-step guide
● Interactive activities to practice the steps
○ Participation through Google docs
Questions?
Data Distancing Series:Systematic Reviews & Meta-Analyses Part 2
Dr. Michal Perlman, Samantha Burns & Gabriella Nocita May 15, 2020
Outline – Day 2
1. Steps for Conducting Systematic Reviews & Meta-Analyses
2. Getting Started○ Developing a research question○ Defining key variables○ Determining inclusion criteria
3. Putting it All Together○ Prisma Flow Chart○ Forest Plot
4. General Conclusions
Steps for Conducting Systematic Reviews & Meta-Analyses
Starting a Systematic Review & Meta-Analysis
Developing a Research Question
What is a Research Question?
● The initial step in a research project
● An answerable inquiry into a topic
Systematic Review Research Questions
1. What are the current implemented curriculums that target developing creativity in early childhood education?
1. What are the most important skills required in the 21st century digital economy?
1. What are effective interventions for for reducing nausea and vomiting in adults with cancer receiving chemotherapy?
Meta-Analysis Research Questions1. Is educator professional development associated with child outcomes in
early childhood education?
1. What Interventions are effective for preventing falls in people after stroke?
1. Are Cannabinoids effective for reducing nausea and vomiting in adults with cancer receiving chemotherapy?
Activity #1Create a research question, related to your field, that is appropriate for
both systematic reviews and meta-analyses.
https://docs.google.com/document/d/1VtA4JAQndHtQpc2s48la3YZhXm9KEm-QDmQt9C1n9IQ/edit?usp=sharing
Defining Key Variables
What is a Definition?Conceptual Definition
Describes qualities of the variables independent of time and space
OR: Tells you what the concept means (dictionary definition)
(e.g., Intelligence: the capacity for learning, reasoning and mental activity)
Operational Definition
Clear, concise, detailed and measurable definition of a concept
OR: Tells you how to measure the concept, specifically for your study
(e.g., Intelligence: the total score on a standardized Stanford Binet IQ test)
Deriving Definitions From Research Question
Is educator early childhood specialization associated with child outcomes in early childhood education?
Deriving Definitions From Research Question
Is educator early childhood specialization associated with child outcomes in early childhood education?
Deriving Definitions From Research Question
Is educator early childhood specialization associated with child outcomes in early childhood education?
Definitions of Early Childhood Education
Conceptual
● Out of home care provided to young children by adults
Definitions of Early Childhood Education
Conceptual
● Out of home care provided to young children by adults
Operational
● Licensed, centre-based programs serving preschool-aged children (30-72 months)○ Daycare○ Nursery school○ Preschool○ Pre-kindergarten○ Head Start programs
Deriving Definitions From Research Question
Is educator early childhood specialization associated with child outcomes in early childhood education?
Definitions of Child Outcomes
Conceptual
● Children’s cognitive, academic, socio-emotional, health, and motor development
Definitions of Child Outcomes
Conceptual
● Children’s cognitive, academic, socio-emotional, health, and motor development
Definitions of Child Outcomes
Conceptual
● Children’s cognitive, academic, socio-emotional, health, and motor development
Operational
Academic
Definitions of Child Outcomes
Conceptual
● Children’s cognitive, academic, socio-emotional, health, and motor development
Operational
Academic
Language
Definitions of Child Outcomes
Conceptual
● Children’s cognitive, academic, socio-emotional, health, and motor development
Operational
Academic
Language
Receptive Language
Definitions of Child Outcomes
Conceptual
● Children’s cognitive, academic, socio-emotional, health, and motor development
Operational
Academic
Language
Receptive Language
Scores on Peabody Picture Vocabulary Test (PPVT)
Activity #2Using your research question from Activity #1, determine a conceptual and
operational definition for one of your key terms.
https://docs.google.com/document/d/1R9tlOziBig_B7S1dGkLa1Lk-
dH9c_mKgrNaCOr01iZg/edit?usp=sharing
Determining Inclusion Criteria
SPICE – Synonyms Acronym S P I C E
Meaning Setting Perspective Intervention Comparison Evaluation
Definition
Setting is the context for the question - where
Perspective is the users, potential users, or stakeholders of the service - for whom
Intervention is the action taken for the users, potential users, or stakeholders - what
Comparison is the alternative actions or outcomes -what else
Evaluation is the result or measurement that will determine the success of the intervention - what result or how well
Our Study ECECChildren and Educators
Factors in ECEC Settings
Control or comparison
Effect of Intervention on Creativity, Collaboration and Problem-solving
Examples
ECE, Nursery, Preschool, Headstart, Licensed Centre-based care
People in these settings
Teacher responsiveness, curriculum, materials, activities, environment. Note: Intervention
Business as usual, different programs/intervention
Soft skills, how they are operationalized, effectiveness of interventions on soft skills (or their components)
Putting It Together: Final Product
Prisma Flow Chart
Forest Plot
Conclusions● Systematic reviews and meta-analyses are rigorous ways of synthesizing
research findings
● They offer many benefits, but require careful interpretation
● They save consumers of research lots of time and heartache!
● Being a critical consumer is key
● Many resources are available within and outside of the University of
Toronto to support you in completing a systematic review and meta-
analysis
Thank you!Stay safe and healthy J
[email protected]@utoronto.ca