basic aspects of meta analysis - globaldizajnneuron.mefst.hr/docs/graduate...
TRANSCRIPT
![Page 1: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/1.jpg)
Basic Aspects of Meta‐Analysis
Hannah R. RothsteinBaruch College‐ City University of New
YorkMichael Borenstein
Biostat
Split, Croatia June 2014 1
![Page 2: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/2.jpg)
Agenda
• Primary studies, systematic reviews and meta‐analysis
• Basic components of a meta‐analysis• Effect Sizes• Basic Calculations• Heterogeneity of effects• Comparison of fixed effect and random effects models
Split, Croatia June 2014 2
![Page 3: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/3.jpg)
PRIMARY STUDIES, SYSTEMATIC REVIEWS AND META‐ANALYSIS
Part I
Split, Croatia June 2014 3
![Page 4: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/4.jpg)
Primary study
• Hypothesis• Inclusion/exclusion criteria• Collect data• Statistical analysis (meta‐analysis)• Report
Split, Croatia June 2014 4
![Page 5: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/5.jpg)
Systematic review
• Hypothesis• Inclusion/exclusion criteria• Collect data (information retrieval and data extraction)
• Statistical analysis (meta‐analysis)• Report
Split, Croatia June 2014 5
![Page 6: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/6.jpg)
Meta‐analysis
• Meta‐analysis is the quantitative data analysis component of a systematic review.
• Not all systematic reviews include a meta‐analysis.• Not all meta‐analyses are preceded by the earlier components of a systematic review.
• In practice, researchers will often use “systematic review” , “meta‐analysis” and “research synthesis” as synonyms.
6Split, Croatia June 2014
![Page 7: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/7.jpg)
A Systematic Review Aims To Be
• Explicit (e.g. in its statement of objectives, materials and methods)
• Systematic (e.g. in its identification of literature)
• Transparent (e.g. in its criteria and decisions)• Reproducible (e.g. in its methodology and conclusions)
• Unbiased
7Split, Croatia June 2014
![Page 8: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/8.jpg)
The move to systematic reviews and away from narrative reviews
Split, Croatia June 2014 8
![Page 9: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/9.jpg)
Narrative reviews are not scientifically rigorous
• They use informal and subjective methods to collect and interpret information
• They generally provide a narrative summary of the research literature
• Different experts may perform a review on the same question and come to different conclusions– Sometimes due to review of different sets of studies– But can happen even when the same studies are reviewed
9Split, Croatia June 2014
![Page 10: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/10.jpg)
Narrative reviews become less efficient with more data
• Researcher may be able to combine the results of a few studies in his or her head
• Becomes increasingly difficult to do so as the number of studies increase
• We use statistics to combine information within a single study…….
Split, Croatia June 2014 10
![Page 11: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/11.jpg)
Problems when the treatment effect varies
• As the number of studies grow, they often examine different populations
• The size of the relationship of interest may vary in different populations
• The narrative reviewer, who has enough trouble summarizing studies when they are all done in similar situations now has a much harder task
Split, Croatia June 2014 11
![Page 12: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/12.jpg)
Too much literature
• Among the earliest meta‐analyses were synthesis of:– 345 studies of the effects of interpersonal expectations on behavior (Rosenthal & Rubin, 1978),
– 725 estimates of the relation between class size and academic achievement (Glass & Smith, 1979),
– 833 tests of the effectiveness of psychotherapy (Smith & Glass, 1977),
– 866 comparisons of the differential validity of employment tests for Black and White workers (Hunter, Schmidt & Hunter, 1979)
12Week One Introduction
![Page 13: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/13.jpg)
Is class size related to student achievement?
• By 1978 there were hundreds of studies on this topic
13Week One Introduction
![Page 14: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/14.jpg)
Narrative review
• Thompson concluded that the relationship ...involved too many complex issues to be reduced to a single testable hypothesis, and that research findings were “necessarily inconclusive”.
14Week One Introduction
![Page 15: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/15.jpg)
Meta‐analysis
• Smith and Glass (1978) – 80 studies – 700 comparisons of smaller and larger classes
• Results showed clearly that smaller classes are better on – student achievement,– classroom processes– teacher and student attitudes.
15Week One Introduction
![Page 16: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/16.jpg)
Systematic vs. Narrative Reviews• Scientific approach (models
itself on primary research)• Criteria determined a priori• Comprehensive search for
relevant information• Explicit methods of data
extraction and coding• Meta‐analysis generally be
used to combine data• Replicable
• Influenced by authors’ point of view (bias)
• Author does not need to justify criteria for inclusion
• Search for data does not need to be comprehensive
• Methods not usually specified
• Narrative summary or vote count
• Can’t replicate review
16Split, Croatia June 2014
![Page 17: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/17.jpg)
An attempted “fix”: Vote counting
• Collect a set of studies– Examine the tests of statistical significance – Tally the proportion significant as Yes votes– Tally the proportion not‐significant as No votes – Majority wins– (There are variants with three categories: positive, , negative and non‐significant)
Split, Croatia June 2014 17
![Page 18: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/18.jpg)
Warr and Perry (1982) Psych. Bulletin
Split, Croatia June 2014 18
![Page 19: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/19.jpg)
Vote counting
• Focuses on the statistical significance of the primary studies
• Vote counting treats a nonsignificant p‐value as evidence that an effect is absent. In fact, though, small, moderate, and even large effect sizes may yield a nonsignificant p‐value due to inadequate statistical power.
• Often wrong, due to low power of primary studies
Split, Croatia June 2014 19
![Page 20: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/20.jpg)
Vote‐Counting ‐‐incorrect conclusions • Hedges and Olkin (1980) showed that the power of vote‐counting can not only be lower than that of the studies on which it is based, but can tend toward zero as the number of studies increases.
• With 20 studies of N=30 and an effect of d= .5 a vote count will fail to detect the effect 75% of the time
• In other words, vote counting is not only misleading, it tends to be more misleading as the amount of evidence increases!Split, Croatia June 2014 20
![Page 21: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/21.jpg)
Vote Counting
• Even when correct, doesn’t provide information about the size of effects or the consistency of effects across studies
• Doesn’t give more weight to more precise studies
Split, Croatia June 2014 21
![Page 22: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/22.jpg)
OTHER APPROACHES TO SYNTHESIS
Special Situations: Combined significance test and sign test
Week One Introduction 22
![Page 23: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/23.jpg)
Combined significance test meta‐analysis (Rosenthal’s early attempt)
•Advantage: lies in the increased power of the overall comparison.
–If several tests consistently favor the research question but fail to reach the level of significance, due to small sample sizes, the overall test is more likely to reach significance because the pooled sample size is much larger.– The hypothesis being tested is that the null hypothesis is true in every study. If this hypothesis is rejected, we can conclude that there is at least one study with a non null effect.
• It tells you nothing about the magnitude of effect size(s)
Week One Introduction 23
![Page 24: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/24.jpg)
JCCP 1987 Shoham‐Salomon & Rosenthal, Paradoxical Interventions
![Page 25: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/25.jpg)
Combining p Values• Collect a set of studies• Extract the p values from all of the tests of significance (whether significant or not)
• Compute an overall p value • You obtain the overall strength of evidence that an effect exists
• Important Note: This test does not tell us anything about the value of an overall effect, or its statistical significance‐
• This was popular in the early days of meta‐analysis; Rosenthal recommended it in 1978.
• We have better methods now, but sometimes this is the best we can do.
![Page 26: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/26.jpg)
Combining p values
• Advantages– You can use this if all you have is the results of significance tests
– You can combine p values from any test statistic representing the substantive hypothesis of interest, even if the studies vary in design or analysis
– A p value is a p value is a p value
Split, Croatia June 2014 26
![Page 27: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/27.jpg)
Combining p values
• Disadvantages– Same issues as in primary studies–May have effect that is large but not statistically significant
• particularly in primary studies
–May have effect that is trivial, yet statistically significant
• a particular problem for overall meta‐analysis
Split, Croatia June 2014 27
![Page 28: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/28.jpg)
A last resort: The sign test
• The sign test is used to count the number of studies with findings in one direction compared to the number of findings in the other direction, without regard to whether the findings are statistically significant
• If a treatment is completely ineffective, we expect that half of the studies will fall on each side of the no‐effect line
![Page 29: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/29.jpg)
The sign test
• Tests the hypothesis that the effect sizes from a collection of K independent studies are all zero
• Simply test if proportion of positive results is 50%– If the treatment has an effect, the probability of getting a positive result is greater than .5
– If it has no effect, the probability of getting a positive result is .5
Split, Croatia June 2014 29
![Page 30: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/30.jpg)
Sign test advantages• The sign test is useful when
– No numeric data are provided from studies, but directions of effects are provided
– Numeric data are SO different that they cannot be combined statistically
– Studies are so diverse in their populations or other characteristics that a pooled effect size is meaningless, but studies are addressing a questions sufficiently similar that the direction of effect is meaningful.
• Results can be tested for statistical significance using the standard binomial test
![Page 31: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/31.jpg)
Sign test disadvantages
• Doesn’t incorporate sample size (give more weight to more precise studies)
• Does not provide an estimate of effect size
Split, Croatia June 2014 31
![Page 32: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/32.jpg)
Sign test: Example
• Health‐related quality of life after liver transplantation: a meta‐analysis (Bravata et al., 1999, Liver TranspSurg).
• “Performed a sign test on 49 studies to evaluate the direction (positive or negative) of the effect of transplantation on QOL. ”
• “The sign test showed significant improvement in posttransplantation physical health (P <.0004), sexual functioning (P <.008), daily activities (P <.02), general HRQL (P <.02), and social functioning (P <.05), but not psychological health (P <.08). “
Split, Croatia June 2014 32
![Page 33: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/33.jpg)
COMPONENTS OF A META‐ANALYSIS
Split, Croatia June 2014 33
![Page 34: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/34.jpg)
Split, Croatia June 2014 34
![Page 35: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/35.jpg)
Split, Croatia June 2014 35
![Page 36: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/36.jpg)
Effect size and weight
• Effect size can be based on means, proportions, and so on
• Weights based on amount of information carried by each study
Split, Croatia June 2014 36
![Page 37: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/37.jpg)
Split, Croatia June 2014 37
![Page 38: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/38.jpg)
Summary effect and CI
• Summary effect is weighted mean• Confidence interval based on same statistics as primary study
• Takes account of multiple levels of sampling
Split, Croatia June 2014 38
![Page 39: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/39.jpg)
Split, Croatia June 2014 39
![Page 40: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/40.jpg)
A meta‐analysis is a synthesis
• How we approach the synthesis depends on our research question
• What kinds of studies we include depends on our research question
• The kind of analysis depends on our research question
• The conclusions depend on our research question and the information available
Split, Croatia June 2014 40
![Page 41: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/41.jpg)
Compared to primary study
• Provides a context– The treatment effect is similar across the set of studies
– The treatment effect varies in unknown ways– The treatment effect varies in ways we can model
Split, Croatia June 2014 41
![Page 42: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/42.jpg)
When the treatment effect is consistent across studies
Meta‐analysis• Provides a more precise estimate of effect size• Provides increased statistical power
Split, Croatia June 2014 42
![Page 43: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/43.jpg)
Streptokinase
• A now classic systematic review and meta‐analysis
Split, Croatia June 2014 43
![Page 44: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/44.jpg)
44Split, Croatia June 2014
![Page 45: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/45.jpg)
Basic Analysis
• Treatment effect was consistent• Combined effect was substantial• Only 6 of 33 studies met criterion for significance
45Split, Croatia June 2014
![Page 46: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/46.jpg)
46Split, Croatia June 2014
![Page 47: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/47.jpg)
Cumulative Analysis
• Meta‐analysis in 1977 could have been definitive
• 40,000 patients randomized after 1977• Additional millions not treated• Even in 1992, narrative review was not definitive
47Split, Croatia June 2014
![Page 48: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/48.jpg)
Studies vary in unknown ways
Meta‐analysis• Provides more precise estimate of effect size• Provides increased statistical power• Allows us to measure the heterogeneity of the effect
Split, Croatia June 2014 48
![Page 49: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/49.jpg)
Split, Croatia June 2014 49
![Page 50: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/50.jpg)
Prevalence
• Based on the heterogeneity index (tau‐squared), most of the variation here is real; that is it is not due to chance.
• Prevalence varied from 5% to 40% across studies; the reasons for this heterogeneity were not investigated.
Split, Croatia June 2014 50
![Page 51: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/51.jpg)
Study name Statistics for each study Event rate and 95% CIEvent Lower Upper
rate limit limitPelcovitz, 1996 0.216 0.112 0.376Barakat, 1997 0.090 0.066 0.122Manne, 1998 0.050 0.021 0.115Fuemmeler, 2001 0.400 0.243 0.581Libov, 2002 0.188 0.116 0.288Brown, 2003 0.235 0.157 0.337Kazak, 2004 0.130 0.090 0.184Manne, 2004 0.108 0.064 0.178Kazak, 2005 0.240 0.186 0.304Landolt, 2005 0.214 0.134 0.326Glover 0.310 0.227 0.407
0.180 0.130 0.244-0.50 -0.25 0.00 0.25 0.50
Prevalence of PTSD
Meta Analysis
Study name Statistics for each study Event rate and 95% CI
Event Lower Upper rate limit limit
Pelcovitz, 1996 0.216 0.112 0.376Barakat, 1997 0.090 0.066 0.122Manne, 1998 0.050 0.021 0.115Fuemmeler, 2001 0.400 0.243 0.581Libov, 2002 0.188 0.116 0.288Brown, 2003 0.235 0.157 0.337Kazak, 2004 0.130 0.090 0.184Manne, 2004 0.108 0.064 0.178Kazak, 2005 0.240 0.186 0.304Landolt, 2005 0.214 0.134 0.326Glover 0.310 0.227 0.407
0.180 0.130 0.244
-1.00 -0.50 0.00 0.50 1.00
Prevalence of PTSD
Random-effects model
Study name Statistics for each study Event rate and 95% CI
Event Lower Upper rate limit limit
Pelcovitz, 1996 0.216 0.112 0.376Barakat, 1997 0.090 0.066 0.122Manne, 1998 0.050 0.021 0.115Fuemmeler, 2001 0.400 0.243 0.581Libov, 2002 0.188 0.116 0.288Brown, 2003 0.235 0.157 0.337Kazak, 2004 0.130 0.090 0.184Manne, 2004 0.108 0.064 0.178Kazak, 2005 0.240 0.186 0.304Landolt, 2005 0.214 0.134 0.326Glover 0.310 0.227 0.407
0.178 0.158 0.201
-1.00 -0.50 0.00 0.50 1.00
Pevalence of PTSD
Random-effects model Split, Croatia June 2014 51
![Page 52: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/52.jpg)
Studies vary in ways we can model
Meta‐analysis• Allows us to assess treatment effect (and variation) within subgroups
• Assess us to measure the impact of moderator(s) on the treatment effect
Split, Croatia June 2014 52
![Page 53: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/53.jpg)
Impact of two DrugsOn Weight Loss
Split, Croatia June 2014 53
![Page 54: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/54.jpg)
Split, Croatia June 2014 54
![Page 55: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/55.jpg)
Outcomes matter
• These two obesity drugs have different mechanisms, different side effects and different contraindications.
• If we had looked at one of the side effects as an outcome, we might have a different picture.
Split, Croatia June 2014 55
![Page 56: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/56.jpg)
Split, Croatia June 2014 56
![Page 57: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/57.jpg)
Group byAnalgesic
Study name Statistics for each study Risk ratio and 95% CI
Risk Lower Upper ratio limit limit p-Value
Ibuprofen 1.294 1.119 1.496 0.001
Paracetamol 1.111 1.060 1.165 0.000
Overall 1.127 1.078 1.179 0.000
0.5 1 2
Favours Placebo Favours Caffeine
Caffeine by Analgesic | Relief
Meta Analysis
Split, Croatia June 2014 57
![Page 58: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/58.jpg)
Split, Croatia June 2014 58
![Page 59: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/59.jpg)
Impact of Statins by Smoking
Split, Croatia June 2014 59
![Page 60: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/60.jpg)
Bottom line
• Can limit the analysis to studies that are essentially identical, and get more precise estimate of the common effect
• Can include studies that vary in random ways, assess variation in effect
• Can include studies that vary on potentially important factors, assess the impact of these factors
Split, Croatia June 2014 60
![Page 61: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/61.jpg)
Caution about moderator analysis
• Moderator variables use up power• Too many moderator analyses lead to Type I errors
• Variables are often confounded with each other
• Moderator analyses are always observational–they don’t support causal inferences
Split, Croatia June 2014 61
![Page 62: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/62.jpg)
Effect‐size indices
Split, Croatia June 2014 62
![Page 63: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/63.jpg)
Indices
• Continuous Data• Dichotomous (Binary) Data• Correlations• Others (E.g. prevalence, mean in one group)
Split, Croatia June 2014 63
![Page 64: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/64.jpg)
Indices for means
• Raw mean difference D• Standardized mean difference d and g
Split, Croatia June 2014 64
![Page 65: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/65.jpg)
Raw mean difference
1 2D X X= −
550 500 50D = − =
Split, Croatia June 2014 65
![Page 66: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/66.jpg)
Standardized Mean Difference
1 2
Within
X XdS−
=
550 500 0 50100
.d −= =
Split, Croatia June 2014 66
![Page 67: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/67.jpg)
Raw vs. Standardized Difference
D d (or g)
Scale is natural or known Required Not required
All studies must use same scale Required Not required
Standard deviation must be consistent
Required Not required
Split, Croatia June 2014 67
![Page 68: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/68.jpg)
Indices for binary outcomes
• Definition of risk• Definition of odds• Risk difference RD• Risk ratio (relative risk) RR• Odds ratio OR
Split, Croatia June 2014 68
![Page 69: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/69.jpg)
Split, Croatia June 2014 69
![Page 70: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/70.jpg)
Meaning of risk
• The number of people with the event/condition we are interested in compared with the total number of people who could have had it.
• Example from Cochrane– 24 people drank Coffee; 6 developed a headache.– The Risk of developing a headache (given coffee drinking) is 6/24 or 25%
Split, Croatia June 2014 70
![Page 71: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/71.jpg)
Meaning of Odds
• The number of people with the event of interest compared with the number without the event of interest.
• In the running Cochrane example, if 24 people drank coffee and 6 developed a headache
• The Odds of developing a headache (given coffee drinking) are 6/18, or 1 in 3.
Split, Croatia June 2014 71
![Page 72: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/72.jpg)
Risk versus Odds
IN GENERAL• When an event is rare, there won’t be too much difference between the risk and the odds
• When an event is common, there can be a BIG difference between the risk and the odds.
Split, Croatia June 2014 72
![Page 73: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/73.jpg)
Risk Difference
• Compute the risk of the event in the treated group
• Compute the risk of the event in the control group
• The Risk difference is the Risk in the treated group MINUS the Risk in the control group
• A Risk difference of ZERO means that there is no difference between the groups
Split, Croatia June 2014 73
![Page 74: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/74.jpg)
Risk Difference
1 2
A CRDn n
⎛ ⎞ ⎛ ⎞= −⎜ ⎟ ⎜ ⎟⎝ ⎠ ⎝ ⎠
5 10 0 05100 100
.RD ⎛ ⎞ ⎛ ⎞= − = −⎜ ⎟ ⎜ ⎟⎝ ⎠ ⎝ ⎠
Split, Croatia June 2014 74
![Page 75: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/75.jpg)
Risk Ratio
• Compute the risk in the treated group• Compute the risk in the control group• The Risk Ratio is the risk in the treated group divided by the risk in the control group
• When the risk ratio is 1, that means there is NO difference between the groups
• When the risk ratio is above 1, it means that the risk is higher in the treated group than in the control group.
Split, Croatia June 2014 75
![Page 76: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/76.jpg)
Risk Ratio (Relative Risk)
1
2
//
A nRRC n
=
5 100 0 5010 100
/ ./
RR = =
Split, Croatia June 2014 76
![Page 77: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/77.jpg)
Risk ratio
5 100 0 5010 100
/ ./
=
10 100 2 005 100
/ ./
=
0 50 2 00 1 252
. . .+=
Split, Croatia June 2014 77
![Page 78: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/78.jpg)
Log Risk ratio
0 50 0 693ln( . ) .= −
2 00 0 693ln( . ) .= +
0 693 0 693 0 002
. . .− +=
Split, Croatia June 2014 78
![Page 79: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/79.jpg)
Split, Croatia June 2014 79
![Page 80: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/80.jpg)
• Compute the odds in the treated group• Compute the odds in the control group• The Odds Ratio is the odds in the treated group divided by the odds in the control group
• When the odds ratio is 1, that means there is NO difference between the groups
Split, Croatia June 2014 80
![Page 81: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/81.jpg)
Odds Ratio
ADORBC
=
5 90 0 4795 10
.OR ×= =
×
Split, Croatia June 2014 81
![Page 82: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/82.jpg)
Split, Croatia June 2014 82
![Page 83: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/83.jpg)
Split, Croatia June 2014 83
![Page 84: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/84.jpg)
“If every diabetic now taking Avandia were instead given a similar pill named Actos, about 500 heart attacks and 300
cases of heart failure would be averted every month”
Split, Croatia June 2014 84
![Page 85: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/85.jpg)
Split, Croatia June 2014 85
![Page 86: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/86.jpg)
“In a highly unusual coordinated announcement, drug regulators in the United
States and Europe said Thursday that Avandia … would no longer be widely available
Split, Croatia June 2014 86
![Page 87: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/87.jpg)
Choice of effect size index: Avandia
• RR is 1.43 – your chances of dying increase by almost 50%
• RD is .005– the risk goes up from one in a thousand, versus 1 ½ per thousand.
Split, Croatia June 2014 87
![Page 88: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/88.jpg)
RD vs. RR vs. ORRD RR OR
Type of difference
Absolute Relative Relative
Intuitive for Researchers Researchers Statisticians
Special statistical issues
Prob. of failure and prob of success are reciprocal
Prob of failure and of success
NOT reciprocal, and weights vary by choice
Prob. of failure and prob of
success are reciprocal
Metric for analyses
Raw Log Log
Split, Croatia June 201488
![Page 89: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/89.jpg)
Cochrane likes the Risk Difference
• It is easy to compute• It is an absolute measure of actual change in risk
• It is easy to convert to natural frequencies and to NNT
• HOWEVER: The Risk Difference is more variable than the RR or OR across different populations, when baseline risk varies.
Split, Croatia June 2014 89
![Page 90: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/90.jpg)
Correlation
• Used when both predictor and outcome variable are continuous.
• Used mostly in observational studies.
Split, Croatia June 2014 90
![Page 91: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/91.jpg)
Split, Croatia June 2014 91
![Page 92: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/92.jpg)
Effect Sizes Conventions
• Uses and cautions
Split, Croatia June 2014 92
![Page 93: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/93.jpg)
Effect sizes rather than p‐values
Split, Croatia June 2014 93
![Page 94: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/94.jpg)
p‐value and effect sizeApproaches are consistent
Split, Croatia June 2014 94
![Page 95: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/95.jpg)
Example‐1
• One study p = .119• One study p = .001
• Which study had the larger effect size?
Split, Croatia June 2014 95
![Page 96: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/96.jpg)
Example 1
Split, Croatia June 2014 96
![Page 97: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/97.jpg)
Example 2
• One study p = .012• One study p = .012• One study p = .012
• Which study had the larger effect size?
Split, Croatia June 2014 97
![Page 98: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/98.jpg)
Example 2
Split, Croatia June 2014 98
![Page 99: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/99.jpg)
Example 3
• One study p = .057• One study p = .035
• Which study had the larger effect size?
Split, Croatia June 2014 99
![Page 100: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/100.jpg)
Example 3
Split, Croatia June 2014 100
![Page 101: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/101.jpg)
Key point
• Always important to work with ES• In meta‐analysis, especially so
Split, Croatia June 2014 101
![Page 102: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/102.jpg)
Key point
• p‐value is poor surrogate for effect size• In primary studies, should report ES• In meta‐analyses must work with ES
Split, Croatia June 2014 102
![Page 103: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/103.jpg)
Three studies, non significant
• Focus on p‐values, synthesize conclusions and might conclude is no evidence of effect
• Focus on effect size, synthesize effects, as in next slide
Split, Croatia June 2014 103
![Page 104: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/104.jpg)
FIXED‐EFFECT META‐ANALYSIS
Meta‐Analysis with consistent effects
Split, Croatia June 2014 104
![Page 105: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/105.jpg)
Sampling errorwhen studies share a common effect size
Split, Croatia June 2014 105
![Page 106: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/106.jpg)
( )11/W V=
Weightswhen studies share a common effect size
Split, Croatia June 2014 106
![Page 107: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/107.jpg)
Split, Croatia June 2014 107
![Page 108: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/108.jpg)
Split, Croatia June 2014 108
![Page 109: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/109.jpg)
Split, Croatia June 2014 109
![Page 110: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/110.jpg)
Split, Croatia June 2014 110
![Page 111: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/111.jpg)
Precision of themean effect
Split, Croatia June 2014 111
![Page 112: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/112.jpg)
Split, Croatia June 2014 112
![Page 113: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/113.jpg)
Impact on precision
Split, Croatia June 2014 113
![Page 114: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/114.jpg)
Split, Croatia June 2014 114
![Page 115: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/115.jpg)
Split, Croatia June 2014 115
![Page 116: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/116.jpg)
Split, Croatia June 2014 116
![Page 117: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/117.jpg)
Split, Croatia June 2014 117
![Page 118: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/118.jpg)
Split, Croatia June 2014 118
![Page 119: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/119.jpg)
Impact on p‐value
Split, Croatia June 2014 119
![Page 120: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/120.jpg)
Split, Croatia June 2014 120
![Page 121: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/121.jpg)
Split, Croatia June 2014 121
![Page 122: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/122.jpg)
Split, Croatia June 2014 122
![Page 123: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/123.jpg)
Split, Croatia June 2014 123
![Page 124: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/124.jpg)
Split, Croatia June 2014 124
![Page 125: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/125.jpg)
Split, Croatia June 2014 125
![Page 126: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/126.jpg)
Split, Croatia June 2014 126
![Page 127: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/127.jpg)
Heterogeneity in effect sizes
Split, Croatia June 2014 127
![Page 128: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/128.jpg)
What we mean by Heterogeneity
It’s the variance in true effects (not observed effects) that we care about
Split, Croatia June 2014 128
![Page 129: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/129.jpg)
Why estimate heterogeneity?
Split, Croatia June 2014 129
![Page 130: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/130.jpg)
It affects the weights
Split, Croatia June 2014 130
![Page 131: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/131.jpg)
Weights when T2 = 0
( )11/W V=
Split, Croatia June 2014 131
![Page 132: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/132.jpg)
Weights when T2 > 0
( )211/W V T= +
Split, Croatia June 2014 132
![Page 133: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/133.jpg)
It affects the mean
Split, Croatia June 2014 133
![Page 134: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/134.jpg)
It affects the standard error …
… the confidence interval, and the p‐value
Split, Croatia June 2014 134
![Page 135: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/135.jpg)
It affects the utility of the treatment
Is the treatment effective for everyone, or effective for some and harmful for others?
Split, Croatia June 2014 135
![Page 136: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/136.jpg)
Impact on Standard Error
Split, Croatia June 2014 136
![Page 137: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/137.jpg)
Split, Croatia June 2014 137
![Page 138: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/138.jpg)
Split, Croatia June 2014 138
![Page 139: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/139.jpg)
Split, Croatia June 2014 139
![Page 140: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/140.jpg)
Split, Croatia June 2014 140
![Page 141: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/141.jpg)
Precision of themean effect
Split, Croatia June 2014 141
![Page 142: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/142.jpg)
Precision of themean effect
Split, Croatia June 2014 142
![Page 143: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/143.jpg)
Precision of themean effect
Split, Croatia June 2014 143
![Page 144: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/144.jpg)
Impact on mean
• As heterogeneity increases, mean effect shifts away from larger studies and towards smaller studies
Split, Croatia June 2014 144
![Page 145: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/145.jpg)
Split, Croatia June 2014 145
![Page 146: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/146.jpg)
Split, Croatia June 2014 146
![Page 147: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/147.jpg)
Impact on substantive utility
Split, Croatia June 2014 147
![Page 148: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/148.jpg)
Split, Croatia June 2014 148
![Page 149: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/149.jpg)
Split, Croatia June 2014 149
![Page 150: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/150.jpg)
Split, Croatia June 2014 150
![Page 151: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/151.jpg)
Split, Croatia June 2014 151
![Page 152: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/152.jpg)
Split, Croatia June 2014 152
![Page 153: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/153.jpg)
Split, Croatia June 2014 153
![Page 154: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/154.jpg)
Split, Croatia June 2014 154
![Page 155: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/155.jpg)
Dispersion of theindividual effects
Split, Croatia June 2014 155
![Page 156: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/156.jpg)
Dispersion of theindividual effects
Split, Croatia June 2014 156
![Page 157: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/157.jpg)
That’s why we need to quantify
Split, Croatia June 2014 157
![Page 158: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/158.jpg)
How do we quantify heterogeneity?
Split, Croatia June 2014 158
![Page 159: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/159.jpg)
Two‐step process
• Isolate the real dispersion• Translate this into useful indices
Split, Croatia June 2014 159
![Page 160: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/160.jpg)
Part 1 – Isolate the real dispersion
Split, Croatia June 2014 160
![Page 161: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/161.jpg)
What we’d like to see if the true effect is the same in all studies
Split, Croatia June 2014 161
![Page 162: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/162.jpg)
What we might see if the true effect is the same in all studies
Split, Croatia June 2014 162
![Page 163: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/163.jpg)
Is there real dispersion?
Split, Croatia June 2014 163
![Page 164: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/164.jpg)
It depends on the precision
Split, Croatia June 2014 164
![Page 165: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/165.jpg)
It depends on the precision
Split, Croatia June 2014 165
![Page 166: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/166.jpg)
Key point
• We can easily compute variance of observedeffects
• But this is due partly to real differences in effects and partly to sampling error within studies
• We need to isolate the between‐studies variance
Split, Croatia June 2014 166
![Page 167: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/167.jpg)
Split, Croatia June 2014 167
![Page 168: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/168.jpg)
Split, Croatia June 2014 168
![Page 169: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/169.jpg)
Split, Croatia June 2014 169
![Page 170: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/170.jpg)
To assess heterogeneity
• Compute observed variance• Estimate how much variance would be expected if true effect is identical in all studies
• Observed minus expected is estimate of true variance
Split, Croatia June 2014 170
![Page 171: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/171.jpg)
Isolating the real dispersion
Q df Q‐df
Split, Croatia June 2014 171
![Page 172: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/172.jpg)
Indices related to heterogeneity
Q−df Basis for all indicesp Test of nullT Standard deviation of true effectsT2 Variance of true effectsI2 Proportion of true/total variance
Split, Croatia June 2014 172
![Page 173: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/173.jpg)
P‐value
• Can we conclude that there is some variance in true effects
• Depends on amount of excess variance andthe amount of evidence
Split, Croatia June 2014 173
![Page 174: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/174.jpg)
Split, Croatia June 2014 174
![Page 175: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/175.jpg)
Split, Croatia June 2014 175
![Page 176: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/176.jpg)
Split, Croatia June 2014 176
![Page 177: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/177.jpg)
Statistics apply to both models
Split, Croatia June 2014 177
![Page 178: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/178.jpg)
Caution !
• I2 is NOT a measure of absolute heterogeneity• I2 tells us what proportion of the observed dispersion reflects differences in true scores rather than random sampling error
Split, Croatia June 2014 178
![Page 179: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/179.jpg)
What proportion of the observed variance is real?
I2
Split, Croatia June 2014 179
![Page 180: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/180.jpg)
Fixed‐effect vs. Random‐effects
Split, Croatia June 2014 180
![Page 181: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/181.jpg)
Fixed‐effect vs. Random‐effects
• Fixed‐effect– Sampling takes place at one level only– Any between‐study variance will be ignored when assigning weights
• Random‐effects– Sampling takes place at two levels– Any between‐study variance will be used when assigning weights
Split, Croatia June 2014 181
![Page 182: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/182.jpg)
Fixed effect
• When there is reason to believe that all the studies are functionally identical
• When our goal is to compute the common effect size, for the studies in the analysis
• Example of drug company has run five studies to assess the effect of a drug.
Split, Croatia June 2014 182
![Page 183: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/183.jpg)
Random effects
• When not likely that all the studies are functionally equivalent.
• When the goal of this analysis is to generalize to a range of populations.
• Example of studies culled from publications
Split, Croatia June 2014 183
![Page 184: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/184.jpg)
Split, Croatia June 2014 184
![Page 185: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/185.jpg)
Split, Croatia June 2014 185
![Page 186: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/186.jpg)
Definition of combined effect
• Fixed effect model– There is one true effect– Summary effect is estimate of this value
• Random effects model– There is a distribution of effects– Summary effect is mean of distribution
Split, Croatia June 2014 186
![Page 187: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/187.jpg)
How weights are assigned
Split, Croatia June 2014 187
![Page 188: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/188.jpg)
Fixed effect model
( )11/W V=
Split, Croatia June 2014 188
![Page 189: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/189.jpg)
Random‐effects model
( )211/W V T= +
Split, Croatia June 2014 189
![Page 190: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/190.jpg)
How weights shift
• If within‐study variance only, W=1/V• If between‐study variance only, W=1/T2
• If both, W=1/(V+T2)
Split, Croatia June 2014 190
![Page 191: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/191.jpg)
Split, Croatia June 2014 191
![Page 192: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/192.jpg)
Split, Croatia June 2014 192
![Page 193: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/193.jpg)
Random vs. Fixed
Split, Croatia June 2014 193
![Page 194: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/194.jpg)
Random vs. Fixed
RE weights are more balanced
Split, Croatia June 2014 194
![Page 195: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/195.jpg)
Random vs. Fixed
RE confidence interval is wider
Split, Croatia June 2014 195
![Page 196: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/196.jpg)
Large study has less impact under RE
Split, Croatia June 2014 196
![Page 197: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/197.jpg)
Small study has more impact under RE
Split, Croatia June 2014 197
![Page 198: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/198.jpg)
FESplit, Croatia June 2014 198
![Page 199: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/199.jpg)
Summary effect
101 171 0 414244 215
. .
.=FE
Split, Croatia June 2014 199
![Page 200: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/200.jpg)
Variance of summary effect
1 0 004244 215
..
=FESplit, Croatia June 2014 200
![Page 201: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/201.jpg)
RESplit, Croatia June 2014 201
![Page 202: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/202.jpg)
Summary effect
32 342 0 35890 284
. .
.=RE
Split, Croatia June 2014 202
![Page 203: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/203.jpg)
Variance of summary effect
1 0 01190 284
..
=RESplit, Croatia June 2014 203
![Page 204: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/204.jpg)
Why does it matter?
• One matches the sampling• One does not• Wrong model yields incorrect weights• Estimate of mean is wrong• Estimate of CI is wrong
Split, Croatia June 2014 204
![Page 205: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/205.jpg)
Fixed vs. Random
• MUST choose based on sampling model• The meaning of the ES is different• Relative weights are closer under RE (effect size will shift)
• Absolute weights are smaller under RE (CI will become wider)
• p‐value will change (less significant in long run but can go either way)
Split, Croatia June 2014 205
![Page 206: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/206.jpg)
Test of null
=M
MZSE
=*
*
*M
MZSE
Fixed
RandomTwo sources of error
Onesource of error
Split, Croatia June 2014 206
![Page 207: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/207.jpg)
Question!
Suppose we had four studies, each with N =1,000,000, and a true (mean) effect size of 0.50. Under the two models,
– What would the forest plot look like?– What would the diamond look like?
Split, Croatia June 2014 207
![Page 208: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/208.jpg)
MSEk n
2
=×σ
Split, Croatia June 2014 208
![Page 209: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/209.jpg)
MTSE
k n k
2 2
* = +×σ
Split, Croatia June 2014 209
![Page 210: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/210.jpg)
Statistical power, Fixed‐effect
d
dPower fSE
⎛ ⎞= ⎜ ⎟
⎝ ⎠
2
dσSEk n
=×
Split, Croatia June 2014 210
![Page 211: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/211.jpg)
Statistical power, Random‐effects
d
dPower fSE
⎛ ⎞= ⎜ ⎟
⎝ ⎠
2 2
dσ τSEk n k
= +×
Split, Croatia June 2014 211
![Page 212: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/212.jpg)
Need to know the source of the variance
• If one source, then error is V/n• If two sources, then error is V/n + T2/k
Split, Croatia June 2014 212
![Page 213: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/213.jpg)
Which model should we use?
• Base decision on the model that matched the way the data were collected
• Not on test of homogeneity
Split, Croatia June 2014 213
![Page 214: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/214.jpg)
Split, Croatia June 2014 214
![Page 215: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/215.jpg)
What you may hear
• Fixed‐effect is simple model• Random‐effects is more complicated
Split, Croatia June 2014 215
![Page 216: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/216.jpg)
Actually
• Fixed‐effect is more restricted model• Random‐effects makes less assumptions
Split, Croatia June 2014 216
![Page 217: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/217.jpg)
An alternate view
• Random‐effects model only makes sense if we have a clear picture of the sampling frame
• Otherwise, we should report the mean and CI for the studies in our sample without attempt to generalize to a larger universe
• This is a fixed‐effects analysis (in the plural) where “fixed” means “set” rather than “common”
Split, Croatia June 2014 217
![Page 218: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/218.jpg)
Prediction intervals
Split, Croatia June 2014 218
![Page 219: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/219.jpg)
Precision of themean effect
Dispersion of theindividual effects
Split, Croatia June 2014 219
![Page 220: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/220.jpg)
Confidence and Prediction
Split, Croatia June 2014 220
![Page 221: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/221.jpg)
Confidence Interval
• Measure of precision• Range for the true value of the mean• Analogous to standard error• Applies to fixed or random effects model
Split, Croatia June 2014 221
![Page 222: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/222.jpg)
Prediction Interval
• Measure of dispersion• Range for the true effect in different studies• Analogous to standard deviation
Split, Croatia June 2014 222
![Page 223: Basic Aspects of Meta Analysis - Globaldizajnneuron.mefst.hr/docs/graduate school/tribe/workshops/metaanalize... · Basic Aspects of Meta‐Analysis Hannah R. Rothstein Baruch College‐City](https://reader031.vdocuments.mx/reader031/viewer/2022022012/5b16d1987f8b9a776d8de41d/html5/thumbnails/223.jpg)
Confidence Interval vs.Prediction Interval
Split, Croatia June 2014 223