what is meta-analysis?
DESCRIPTION
What is meta-analysis?. ESRC Research Methods Festival Oxford 8 th July, 2010 Professor Steven Higgins Durham University [email protected]. Key points. Understanding ‘effect-size’ Why do we need it? What are its limitations? What is its potential?. What is an “effect size”?. - PowerPoint PPT PresentationTRANSCRIPT
What is meta-analysis?
ESRC Research Methods FestivalOxford
8th July, 2010
Professor Steven HigginsDurham University
Key points Understanding ‘effect-size’ Why do we need it? What are its limitations? What is its potential?
What is an “effect size”?
Standardised way of looking at difference Different methods for calculation
Odds Ratio Correlational (Pearson’s r) Standardised mean difference
Difference between control and intervention group as proportion of the dispersion of scores
Examples of Effect Sizes:
ES = 0.2“Equivalent to the difference in heights between 15 and 16 year old girls”
58% of
control group below
mean of experimental
group
Probability you could guess which group a person was in = 0.54Change in the proportion above a given threshold:
from 50% to 58% or from 75% to 81%
“Equivalent to the difference in heights between 14 and 18 year old girls”
69% of
control group below
mean of experimental
group
Probability you could guess which group a person was in = 0.60
ES = 0.5
Change in the proportion above a given threshold:from 50% to 69% or from 75% to 88%
“Equivalent to the difference in heights between 13 and 18 year old girls”
79% of
control group below
mean of experimental
group
Probability you could guess which group a person was in = 0.66
ES = 0.8
Change in the proportion above a given threshold:
from 50% to 79% or from 75% to 93%
Significance versus effect size Traditional test is of statistical ‘significance’ The difference is unlikely to have occurred by
chance However it may not be:
Large Important, or even Educationally ‘significant’
The rationale for using effect sizes Traditional reviews focus on statistical significance
testing Highly dependent on sample size Null finding does not carry the same “weight” as a
significant finding Meta-analysis focuses on the direction and
magnitude of the effects across studies From “Is there a difference?” to “How big is the
difference?” and “How consistent is the difference?” Direction and magnitude represented by “effect size”
Meta-analysis Synthesis of quantitative data
Cumulative Comparative Correlational
“Surveys” educational research (Lipsey and Wilson, 2001)
Forest plots Effective way of presenting results
Studies, effect sizes, confidence intervals Provides an overview of consistency of effects Summarises an overall effect (with confidence
interval) Useful visual model of a meta-analysis
Anatomy of a forest plot…
Studies
N of studyLine of no effect
C.I
Study effect size
Pooled effect size
Pooled effect size
Study effect size (with C.I.)
Weighting of study in meta-analysis
Issues and challenges in meta-analysis
Conceptual Reductionist - the answer is 42 Comparability - apples and oranges Atheoretical - ‘flat-earth’
Technical Heterogeneity Publication bias Methodological quality
Schulze, R. (2007) The state and the art of meta-analysis Zeitschrift fur Psychologie/ Journal of Psychology, 215 pp 87 - 89.
RDI in Quantitative Synthesis Collaboration between the Universities of
Durham, Birmingham and the Institute of Education, University of London
Rob Coe, Mark Newman, James Thomas and Carole Torgerson
Levels 1 and 2: Intro and practical training Durham, Edinburgh London, Belfast, York, Cardiff
Level 3: expert workshops Prof Mark Lipsey, Prof Larry Hedges,
Doctoral support through BERA
References, further readings and information
Books and articlesBorenstein, M., Hedges, L.V., Higgins, J.P.T. & Rothstein, H.R. (2009) Introduction to Meta Analysis (Statistics in Practice) Oxford: Wiley
Blackwell.Chambers, E.A. (2004). An introduction to meta-analysis with articles from the Journal of Educational Research (1992-2002). Journal of
Educational Research, 98, pp 35-44.Cooper, H.M. (1982) Scientific Guidelines for Conducting Integrative Research Reviews Review Of Educational Research 52; 291.*Cooper, H.M. (2009) Research Synthesis and meta-analysis: a step-by-step approach London: SAGE Publications (4th Edition).Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R.O., Hornik, R. C., Phillips, D. C., Walker, D. F., & Weiner, S. S. (1980). Toward
reform of program evaluation: Aims, methods, and institutional arrangements. San Francisco, Ca.: Jossey-Bass. Glass, G.V. (2000). Meta-analysis at 25. Available at: http://glass.ed.asu.edu/gene/papers/meta25.html (accessed 9/9/08)Lipsey, Mark W., and Wilson, David B. (2001). Practical Meta-Analysis. Applied Social Research Methods Series (Vol. 49). Thousand Oaks,
CA: SAGE Publications.Torgerson, C. (2003) Systematic Reviews and Meta-Analysis (Continuum Research Methods) London: Continuum Press.
WebsitesWhat is an effect size?, by Rob Coe: http://www.cemcentre.org/evidence-based-education/effect-size-resources The meta-analysis of research studies: http://echo.edres.org:8080/meta/The Meta-Analysis Unit, University of Murcia: http://www.um.es/metaanalysis/The PsychWiki: Meta-analysis: http://www.psychwiki.com/wiki/Meta-analysis Meta-Analysis in Educational Research: http://www.dur.ac.uk/education/meta-ed/