conducting a meta-analysis using cma: an introduction...mar 25, 2017 · conducting a meta-analysis...
TRANSCRIPT
Conducting a Meta-Analysis Using CMA:
An Introduction
Alison C. Koenka, Ph.D.
March 25, 2017
2
primary studies
3
introducing…meta-analysis!
Dent & Koenka, 2016
4
• Foundation
overview
5
• Foundation
- Formal definition
- Steps in conducting a meta-analysis
- Common questions confronted
overview
6
• Foundation
• Introducing Comprehensive Meta-Analysis
(CMA)
• Inputting data into CMA
overview
7
• Foundation
• Introducing Comprehensive Meta-Analysis
(CMA)
• Inputting data into CMA
• Computing the overall effect size
• Moderator analyses
overview
8
• Foundation
• Introducing Comprehensive Meta-Analysis
(CMA)
• Inputting data into CMA
• Computing the overall effect size
• Moderator analyses
• Advanced topics and additional resources
overview
9
• Foundation
overview
10
• Research synthesis
formal definition
11
formal definition
Literature Review
Research Synthesis Theoretical Review
Meta-
Analysis
Systematic
Review
12
formal definition
Literature Review
Research Synthesis Theoretical Review
Meta-
Analysis
Systematic
Review
13
• Research synthesis
• Statistical integration of study outcomes
formal definition
14
• Research synthesis
• Statistical integration of study outcomes
• Effect sizes:
- Cohen’s d index
- Correlation coefficient (r)
- Odds ratio
formal definition
15
• Formulating the problem
• Searching the literature
• Retrieving information from studies
• Integrating study outcomes
steps in conducting a meta-analysis
16
• Formulating the problem
• Searching the literature
• Retrieving information from studies
• Integrating study outcomes
steps in conducting a meta-analysis
17
• Integrating study outcomes
• Two main components
1. Computing overall effect size
2. Conducting moderator analyses
steps in conducting a meta-analysis
18
Which effect size metric should I use?
common questions confronted
19
• d-index
- Intervention
- Experimental/quasi-experimental
common questions confronted
20
• d-index
• Correlation coefficient (r)
- Correlational data
common questions confronted
21
• d-index
• Correlation coefficient (r)
• Odds ratio
- Dichotomous outcomes
common questions confronted
22
How do I address the issue of publication bias?
common questions confronted
23
• Trim-and-fill procedure
- One of several options
common questions confronted
Duval & Tweedie (2000a; 2000b)
24
• Trim-and-fill procedure
common questions confronted
Duval & Tweedie (2000a; 2000b)
25
How do I identify independent comparisons
(and why is this important)?
common questions confronted
26
• Shifting unit of analysis approach
- One of many options
common questions confronted
Cooper (2016)
27
• Shifting unit of analysis approach
- One of many options
- Implications for importing data into CMA
common questions confronted
Cooper (2016)
28
Which model of error should I use?
common questions confronted
29
• Two options:
1. Fixed-effect model of error
common questions confronted
Bornstein et al. (2009)
30
• Two options:
1. Fixed-effect model of error:
All studies share a common effect size
common questions confronted
Bornstein et al. (2009)
31
• Two options:
1. Fixed-effect model of error:
All studies share a common effect size
Differences solely due to sampling error
common questions confronted
Bornstein et al. (2009)
32
• Two options:
1. Fixed-effect model of error
2. Random-effects model of error
common questions confronted
Bornstein et al. (2009)
33
• Two options:
1. Fixed-effect model of error
2. Random-effects model of error:
Studies do not share a common effect size
common questions confronted
Bornstein et al. (2009)
34
• Two options:
1. Fixed-effect model of error
2. Random-effects model of error:
Studies do not share a common effect size
Differences due to sampling error and true differences
common questions confronted
Bornstein et al. (2009)
35
• Foundation
• Introducing Comprehensive Meta-Analysis
(CMA)
• Inputting data into CMA
overview
36
introducing CMA
37
introducing CMA
38
• Pros:
- Extremely accessible
- Easy to interpret output
introducing CMA
39
• Pros:
- Extremely accessible
- Easy to interpret output
• Cons:
- Perhaps less flexible
- Very expensive
introducing CMA
40
• Three different versions
introducing CMA
www.meta-analysis.com
41
• Step 1: identify column names
- Identify column for
study name
effect size data
inputting data into CMA
42
• Step 1: identify column names
- Effect size data:
Click “next”
Choose default (“comparison of two groups”)
Choose “continuous (means)” for d-index,
Choose “dichotomous” for odds ratio,
OR
Choose “correlation” for r
inputting data into CMA
43
• Step 1: identify column names
- Effect size data (continued):
Choose “continuous (means)” for d-index,
Choose “Cohen’s d option”
Choose effect size data columns
Group A: treatment group
Group B: control group
inputting data into CMA
44
inputting data into CMA
45
• Major caveat of standard version:
- No “subgroup within study” column option
inputting data into CMA
46
• Open CMA and practice these steps
inputting data into CMA
47
• Five pieces of information required:
- Study name
- Effect size counter
- Effect size
- n for ‘treatment’ and ‘comparison’ group
preparing data for CMA
48
• Preparing ‘study name’ and ‘tricking’ CMA
- Necessary for proper treatment of
independent (and non-independent) effect sizes
preparing data for CMA
49
• Preparing ‘study name’ and ‘tricking’ CMA
preparing data for CMA
50
• Preparing ‘study name’ and ‘tricking’ CMA
preparing data for CMA
51
• Preparing ‘study name’ and ‘tricking’ CMA
preparing data for CMA
52
• Preparing ‘study name’ and ‘tricking’ CMA
preparing data for CMA
53
• Foundation
• Introducing Comprehensive Meta-Analysis
(CMA)
• Inputting data into CMA
• Computing the overall effect size
• Moderator analyses
overview
54
• Select “analyses” -- > run analyses
computing the overall effect size
55
• Select “analyses” -- > run analyses
- Select model(s) of error
computing the overall effect size
56
• Select “analyses” -- > run analyses
- Select model(s) of error
- Select *study* and not subgroup as unit of analysis
Select “computational options”
Select “select by…”
Click “subgroups” tab
Use study as the unit of analysis
Click “apply”
computing the overall effect size
57
• Select “analyses” -- > run analyses
- Select model(s) of error
- Select *study* and not subgroup as unit of analysis
computing the overall effect size
This output
incorrectly
uses the
subgroup as
the unit of
analysis,
treating effect
sizes as
independent
58
• Select “analyses” -- > run analyses
- Select model(s) of error
- Select *study* and not subgroup as unit of analysis
computing the overall effect size
This output
correctly
uses the study
as the unit of
analysis,
treating effect
sizes from the
same sample
as dependent
59
• Select “next table”
• Practice these steps
computing the overall effect size
60
• Select “next table”
• This is where you interpret your output
computing the overall effect size
Overall,
average
d-index
61
• Select “next table”
• This is where you interpret your output
computing the overall effect size
Indicates
whether there is
significant
variation
surrounding
average ES
62
• Identify column for….moderator variable
- Name variable (e.g., “grade level”)
- Select “categorical” for data type
conducting a moderator analysis
63
• Identify column for….moderator variable
• Paste data from Excel
conducting a moderator analysis
64
• Identify column for….moderator variable
• Paste data from Excel
• Same steps to run analyses, and
conducting a moderator analysis
65
• Identify column for….moderator variable
• Paste data from Excel
• Same steps to run analyses, and
• Computational options group by…
conducting a moderator analysis
66
• Identify column for….moderator variable
• Paste data from Excel
• Same steps to run analyses, and
• Computational options group by…
conducting a moderator analysis
67
• Follow these steps to conduct “grade level”
moderator analysis yourself
• What would we conclude?
conducting a moderator analysis
68
• Interpreting the output
conducting a moderator analysis
69
• Interpreting the output
conducting a moderator analysis
i.e., “random effects”
70
• Interpreting the output
conducting a moderator analysis
Very different
conclusions
depending on
model of error
71
• Interpreting the output
• Because of small k, will substantively interpret fixed
conducting a moderator analysis
Bornstein et al. (2009)
72
• A note on shifting unit of analysis approach
- Often need to recode study names
conducting a moderator analysis
73
• A note on shifting unit of analysis approach
conducting a moderator analysis
74
• Foundation
• Introducing Comprehensive Meta-Analysis
(CMA)
• Inputting data into CMA
• Computing the overall effect size
• Moderator analyses
• Advanced topics and additional resources
overview
75
• Option under “analysis” (but not in all versions)
publication bias
76
• Interpreting “trim and fill” output
publication bias
Duval & Tweedie (2000a; 2000b)
77
• Interpreting “trim and fill” output
publication bias
78
• Interpreting “trim and fill” output
publication bias
79
• Meta-regression
other advanced topics
Bornstein et al. (2009)
80
• Meta-regression
• Small sample size correction
other advanced topics
Bornstein et al. (2009)
81
• Meta-regression
• Small sample size correction
• Data from cluster-randomized studies
other advanced topics
Hedges (2009)
82
additional resources
Bornstein et al. (2009)
83
• Effect size calculator
additional resources
www.campbellcollaboration.org
84
• Online courses using CMA offered
www.statistics.com
additional resources
85
• CMA practice
www.meta-analysis.com
additional resources
86
• CMA troubleshooting
www.meta-analysis.com
me!
additional resources
