meta analysis

54
A Workshop on the Basics of A Workshop on the Basics of Systematic Review & Systematic Review & Meta-Analysis Meta-Analysis Philip C. Abrami, Robert M. Bernard Philip C. Abrami, Robert M. Bernard C. Anne Wade, Evgueni Borokhovski, Rana Tamim, C. Anne Wade, Evgueni Borokhovski, Rana Tamim, Gretchen Lowerison & Mike Surkes Gretchen Lowerison & Mike Surkes Centre for the Study of Learning and Performance Centre for the Study of Learning and Performance and CanKnow and CanKnow Concordia University Concordia University

Upload: aniuskmarin

Post on 26-May-2015

579 views

Category:

Education


1 download

TRANSCRIPT

Page 1: meta analysis

A Workshop on the Basics ofA Workshop on the Basics ofSystematic Review & Systematic Review &

Meta-Analysis Meta-Analysis

Philip C. Abrami, Robert M. BernardPhilip C. Abrami, Robert M. Bernard

C. Anne Wade, Evgueni Borokhovski, Rana Tamim, C. Anne Wade, Evgueni Borokhovski, Rana Tamim, Gretchen Lowerison & Mike SurkesGretchen Lowerison & Mike Surkes

Centre for the Study of Learning and Performance Centre for the Study of Learning and Performance

and CanKnowand CanKnow

Concordia UniversityConcordia University

Page 2: meta analysis

02/25/11 2

What is a Systematic Review?What is a Systematic Review?

• A review of a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review.

• Statistical methods (meta-analysis) may or may not be used to analyze and summarize the results of the included studies.

• Other examples: Narrative review, qualitative review, vote count, meta-synthesis.

Page 3: meta analysis

02/25/11 3

What is Meta-Analysis?What is Meta-Analysis?

• Meta-Analysis is a set of

quantitative research synthesis

techniques and procedures• Meta-Analysis uses effect size as a

metric for judging the magnitude of standardized difference between a treatment and control condition

Page 4: meta analysis

02/25/11

Page 5: meta analysis

02/25/11 5

Purpose: Purpose: Explaining Explaining Variability in Effect SizeVariability in Effect Size

Effect SizesStudy Features

Shared Variability

Unique Variability Unique Variability

Prediction

Page 6: meta analysis

02/25/11 6

1. Determine the research question2. Develop terms and definitions related to the question3. Develop a search strategy for identification of relevant

studies4. Establish criteria for inclusion and exclusion of studies5. Select studies based on abstract review (agreement)6. Select studies based on full-text review (agreement)7. Extract effect sizes (agreement)8. Develop codebook of study features9. Code studies (agreement)10. Conduct statistical analysis and interpretation

10 Steps in Planning and 10 Steps in Planning and Conducting a Systematic Conducting a Systematic

Review/Meta-AnalysisReview/Meta-Analysis

Page 7: meta analysis

02/25/11

Page 8: meta analysis

02/25/11 8

1. Determine the research question

The “big question” that guides the research. It usually involves asking about the difference

between two conditions (i.e., usually treatment and control) or the relationship

between two measures.

10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

Page 9: meta analysis

02/25/11 9

Questions the Researcher Questions the Researcher Should AskShould Ask

• Does the question have theoretical or practical relevance (i.e., aids in practice and/or policy making decisions)?

• Is the literature of a type that can answer the question?

• Is there a sufficient quantitative research literature?

• Do the studies lend themselves to meta-analysis?

• Is the literature too large given the resources available?

Page 10: meta analysis

02/25/11 10

Example: Example: Critical ThinkingCritical Thinking

Research Question: What instructional interventions, to what extent, and under what particular circumstances, impact on the development and effective use of learner’s critical thinking skills and dispositions?

Page 11: meta analysis

02/25/11 11

2. Develop terms and definitions related to the question

This helps refine the research question and inform the search strategies.

10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

Page 12: meta analysis

02/25/11 12

3. Develop a search strategy for the identification of relevant studies

This involves the planning/implementation of search and retrieval for primary studies (e.g., electronic databases, branching).

10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

Page 13: meta analysis

02/25/11 13

Information Retrieval: Information Retrieval: A Continuous ProcessA Continuous Process

Preliminary Searches Supports beginning steps: Definition of key concepts & research question Use of standard reference tools and broad searches for review articles and key primary studies

Main Searches Identification of primary studies through searches of online databases, printed indices, Internet, branching, hand-searches Most difficult given a number of challenges

Final Searches Occurs towards the end of the Review Process Refine search terms and update original searches

Page 14: meta analysis

02/25/11 14

Preliminary SearchesPreliminary SearchesReference Sources:Purpose: To obtain definitions for the terms; creativity, critical thinking,

decision making, divergent thinking, intelligence; problem solving, reasoning, thinking.

Sources:

Bailin, S. (1998). Critical Thinking: Philosophical Issues. [CD-ROM] Education: The Complete Encyclopedia. Elsevier Science, Ltd.

Barrow, R., & Milburn, G. (1990). A critical dictionary of educational concepts: An appraisal of selected ideas and issues in educational theory and practice (2nd ed.). Hertfordshire, UK: Harvester Wheatsheaf

Colman (2001). Dictionary of Psychology (complete reference to be obtained)

Corsini, R. J. (1999). The dictionary of psychology. Philadelphia, PA: Brunner/Mazel

Dejnoka, E. L., & Kapel, D. E. (1991). American educator’s encyclopedia. Westport, CT: Greenwood Press.

…… (see handout)

Page 15: meta analysis

02/25/11 15

Main Searches: DecisionsMain Searches: Decisions

Selection of Primary Information Retrieval Tools

Scope of search: Which fields should be searched (including all related fields)? Availability of indexing tools: Which tools do we have access to at our institution? Are there others who can perform searches for us? Format of indexing tools: What format are they in (e.g. online, print, web-based)? Date: How far back does the indexing go for each tool? Language: What is the language of the material that is indexed? How can we locate non-English material? Unpublished work: How can we access dissertations, reports, & other grey literature?

Page 16: meta analysis

02/25/11 16

Examples of DatabasesExamples of Databases Education: ERIC, British Education Index, Australian

Education Index, Chinese ERIC, CBCA Education, Education index, Education: A SAGE Full-text Collection

Psychology: PsycINFO, PubMed (Medline), Psychology: A SAGE Full-Text Collection

Sociology: Sociological Abstracts, Contemporary Women’s Issues. Sociology: A SAGE Full-text Collection

Multidisciplinary: EBSCO Academic Search Premier, ProQuest Dissertations and Theses Fulltext, FRANCIS, Social Sciences Index, SCOPUS, Web of Science

Page 17: meta analysis

02/25/11 17

Example: Critical ThinkingExample: Critical ThinkingTo date, the following databases have been searched:

• AACE Digital Library (now known as EdITLib)

• ABI/Inform Business • EBSCO Academic Search Premier• ERIC• EconLit• PAIS International• ProQuest Dissertations and Theses Fulltext• PsycINFO• Social Science Index• Sociological Abstracts

Page 18: meta analysis

02/25/11 18

Main Searches: More DecisionsMain Searches: More Decisions

Preparation of Search Strategies What are the key concepts to be searched? How are these represented in each discipline? What are their related terms? How are these key concepts represented in the controlled vocabulary within each database to be searched? (See handout)

Note: these decisions need to be made for each indexing tool used.

Page 19: meta analysis

02/25/11 19

Main Searches: Main Searches: Yet More DecisionsYet More Decisions

Construction of the Search Statements What terms should be searched as descriptors or

as “free text”? What Boolean operators should be used? Where should truncation characters be used?

(e.g. parent* will retrieve parent, parents, parental)

What limiting features are available to narrow results? (e.g. use of Publication Type codes)?

What time period should be searched?

Page 20: meta analysis

02/25/11 20

Example: ERICExample: ERIC

Combining Keywords/Descriptors using Boolean operators:

Searches and records below from: The ERIC Database

#5 #3 and #4 (1520 records)#4 DTC = 142 or DTC = 143 or control group* (322893

records)#3 #1 or #2 (7718 records)#2 critical thinking in DE,ID (7562 records)#1 thinking skills in DE and critical thinking (1269 records)

Page 21: meta analysis

02/25/11 21

Example from our Codebook:ERIC (Date: September 21, 2003; AW)Purpose: To retrieve the first set of abstracts to be reviewed by team according to the current inclusive/exclusion criteria. Result: Hit rate of 514/1520Source code: ERIC1

Searches and records below from: The ERIC Database (1966-2003, June)#5 #3 and #4 (1520 records)#4 DTC = 142 or DTC = 143 or control group* (322893

records)#3 #1 or #2 (7718 records)#2 critical thinking in DE,ID (7562 records)#1 thinking skills in DE and critical thinking (1269 records)

Documenting Your SearchesDocumenting Your Searches

Page 22: meta analysis

02/25/11 22

Next Steps

Repeat these steps for each database to be

searched.(see handout)

Page 23: meta analysis

02/25/11 23

Secondary Retrieval Strategies Locating the grey (unpublished) literature:

- Using the web, & Dissertations Abstracts

Branching: - Scanning the reference section of review articles

Hand searches:- Scanning the Table of Contents of key journals and conference proceedings

Personal contacts:- Contacting key researchers in the field

Main Searches: Main Searches: Yet Still More DecisionsYet Still More Decisions

Page 24: meta analysis

02/25/11 24

Information Retrieval: Information Retrieval: Wrap UpWrap Up

“Shoestring-budget information retrieval is likely to introduce bias, and should be avoided.” (IR Policy Brief,

2004)

Importance of information retrieval process Not a “one-shot”deal Requires expertise in the planning and implementation of searches Library personnel are important members of the team

Use of bibliographic management software Reference Manager, EndNotes, RefWorks

Ability to replicate review Documentation of entire process, including search strategies used for each database, decisions taken, etc.

Page 25: meta analysis

02/25/11 25

10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

4. Establish criteria for inclusion and exclusion of studies

These are the criteria that guide the search for literature and ultimately determine what

studies are in and out of the review.

Page 26: meta analysis

02/25/11 26

Inclusion/Exclusion: Questions

• What characteristics of studies will be used to determine whether a particular effort was relevant to the research question?

• What characteristics of studies will lead to inclusion? exclusion?

• Will relevance decisions be based on a reading of report titles? abstracts? full reports?

• Who will make the relevance decisions?• How will the reliability of relevance decisions

be assessed?

Page 27: meta analysis

02/25/11 27

10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

5. Select studies based on abstract review

This is the initial decision as to what studies will be retrieved as full-text

documents.

Page 28: meta analysis

02/25/11 28

10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

6. Select studies based on full-text review

This is the second decision as to what studies will be included in the review.

Page 29: meta analysis

02/25/11 29

10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

7. Extract effect sizes

Effect sizes extraction involves convertingdescriptive or other statistical information

contained in studies into a standard metric by which studies can be compared.

Page 30: meta analysis

02/25/11 30

What is an Effect size?What is an Effect size?

• A descriptive metric that characterizes the standardized difference (in SD units) between the mean of a control group and the mean of a treatment group (educational intervention)

• Can also be calculated from correlational data derived from pre-experimental designs or from repeated measures designs

Page 31: meta analysis

02/25/11 31

Characteristics of Characteristics of Effect SizesEffect Sizes

• Can be positive or negative • Interpreted as a z-score, in SD unitsSD units, although

individual effect sizes are not part of a z-score distribution

• Can be aggregated with other effect sizes and subjected to other statistical procedures such as ANOVA and multiple regression

• Magnitude interpretation: ≤ 0.20 is a small effect size, 0.50 is a moderate effect size and ≥ 0.80 is a large effect size (Cohen, 1992)

Page 32: meta analysis

02/25/11 32

Effect Size ExtractionEffect Size Extraction

• Effect size extraction is the process of identifying relevant statistical data in a study and calculating an effect size based on those data

• All effect sizes should be extracted by two coders, working independently

• Coders’ results should be compared and a measure of inter-coder agreement calculated and recorded

• In cases of disagreement, coders should resolve the discrepancy in collaboration

Page 33: meta analysis

02/25/11

Page 34: meta analysis

02/25/11

Page 35: meta analysis

02/25/11 35

Example of Example of ESES Extraction with Extraction with Descriptive StatisticsDescriptive Statistics

Study reports: Treatment mean = 42.8 Control Mean = 32.5

Treatment SD = 8.6 Control SD = 7.4

n = 26 n = 31

SDpooled = ((26 - 1)8.62 )+ (31 - 1)7.42 )) / (57 - 2)

SD pooled = (1849 +1642.8) / 55 = 3491.8 / 55 = 63.49 =7.97

d =42.8 - 32.5

7.97=

10.37.97

=1.29

g =d 1 -3

(4(NE + NC ))- 9₩

₩₩

=1.29 1 -

34(26 + 31) - 9

₩₩₩

=1.29 1 -

3219

₩₩₩

=1.27

Procedure: Calculate SDpooled Calculate d and g

Page 36: meta analysis

02/25/11 36

Extracting Effect Sizes in the Extracting Effect Sizes in the Absence of Descriptive StatisticsAbsence of Descriptive Statistics

• Inferential Statistics (t-test, ANOVA, ANCOVA, etc.) when the exact statistics are provided

• Levels of significance, such as p < .05, when the exact statistics are not given (t can be set at the conservative t = 1.96) (Glass, McGaw & Smith,

1981; Hedges, Shymansky & Woodworth, 1989) • Studies not reporting sample sizes for control

and experimental groups should be considered for exclusion

Page 37: meta analysis

02/25/11 37

Examples of Alternative Methods Examples of Alternative Methods

of of ESES Extraction Extraction

d =2tdf

=2(2.56)

63=

5.127.94

=.6448

• Study Reports: t (63) = 2.56, p < .05

• Study Reports: F (1, 63) = 2.56, p < .05

Convert F to t and apply the above equation:

t = F =1.6;df =63

d =2tdf

=2(1.6)7.94

=3.27.94

=.4030

Page 38: meta analysis

02/25/11 38

Zero Effect SizeZero Effect Size

ES = 0.00

Control Condition

Treatment Condition

Overlapping Distributions

Page 39: meta analysis

02/25/11 39

Moderate Effect SizeModerate Effect Size

Control Condition

Treatment Condition

ES = 0.40

Page 40: meta analysis

02/25/11 40

Large Effect SizeLarge Effect Size

Control Condition

Treatment Condition

ES = 0.85

Page 41: meta analysis

02/25/11 41

Mean and VariabilityMean and Variability

Variability

ES+

Note: Results from Bernard, Abrami, Lou, et al. (2004) RER

Page 42: meta analysis

02/25/11 42

10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

8. Develop a codebook

Study feature coding involves describing the relevant characteristics for each study (e.g.,

research methodology, publication source).The codebook details the study

feature categories and their levels.

Page 43: meta analysis

02/25/11 43

Examining Study FeaturesExamining Study Features

• Purpose: to attempt to explain variability in effect size

• Any nominal, ordinal or interval coded study feature can be investigated

• In addition to mean effect size, variability should be investigated

• Study features with small ks may be unstable

Page 44: meta analysis

02/25/11 44

Examples of Study FeaturesExamples of Study Features

• Research methodology• Type and nature of measures• Direction of the statistical test • Publication data• Relevant aspects of the treatment• Relevant aspects of the control

condition

Page 45: meta analysis

02/25/11 45

10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

9. Code studies for study features

Coding study features is perhaps the most time-consuming and onerous aspect of

conducting a meta-analysis. However, it is arguably the most important step because it provides the possibility for

explaining variability in effect sizes.

Page 46: meta analysis

02/25/11 46

10 Steps in a Meta-Analysis10 Steps in a Meta-Analysis

10: Analysis and interpretation

Analysis involves invoking a range of standard statistical tests to examine average effect

sizes, variability and the relationship between study features and effect size. Interpretation is drawing conclusion from these analyses.

Page 47: meta analysis

02/25/11 47

Questions: Questions: Statistical AnalysisStatistical Analysis

• What techniques will be used to combine results of separate tests?

• What techniques will be used to assess and then analyze the variability in findings across studies?

• What sensitivity analyses (i.e., tests of the impact of such decisions on the results of the review) will be carried out and how?

• What statistical procedures will be used to test relationships between study features and effect sizes (e.g., meta regression)

Page 48: meta analysis

02/25/11 48

Homogeneity vs. Heterogeneity Homogeneity vs. Heterogeneity of Effect Sizeof Effect Size

• If homogeneity of effect size is established, then the studies in the meta-analysis can be thought of as sharing the same effect size (i.e., the mean)

• If homogeneity of effect size is violated (heterogeneity of effect size), then no single effect size is representative of the collection of studies (i.e., the “true” mean effect size remains unknown)

Page 49: meta analysis

02/25/11

Page 50: meta analysis

02/25/11 50

Effect size and 95% confidence interval Test of null (2-Tail)Number Studies Point estimate Standard error Variance Lower limit Upper limit Z-value P-value

168 0.34 0.01 0.00 0.31 0.36 23.28 0.00

HeterogeneityQ-value df (Q) P-value1816.71 167.00 0.00

Statistics in Comprehensive Statistics in Comprehensive Meta-Analysis™ Meta-Analysis™

Comprehensive Meta-Analysis 2.0 is a trademark of BioStat®

Interpretation: Moderate ES for all outcomes (g+ = 0.34) in favor of the intervention condition.

Homogeneity of ES is violated. Q-value is significant (i.e., there is too much variability for g+ to represent a true average in the population).

Page 51: meta analysis

02/25/11 51

Examining the Study Examining the Study Feature “Type of Research Design”Feature “Type of Research Design”

g+ = +0.34

OverallEffect

Pre-Post

Designs

Post-Only

Designs

Quasi-Exp.

Designs

Page 52: meta analysis

02/25/11 52

GroupsGroup N of Studies Point estimate Standard error Lower limit Upper limit Q-value df (Q) P-value

one-group 27 0.16 0.04 0.09 0.24 181.30 26.00 0.00post only 87 0.38 0.02 0.34 0.42 651.34 86.00 0.00quasi-exp 54 0.35 0.02 0.31 0.40 957.45 53.00 0.00

Total within 1790.09 165.00 0.00Total between 26.62 2.00 0.00Overall 168 0.34 0.01 0.31 0.36 1816.71 167.00 0.00

Effect size and 95% confidence interval Heterogeneity

Tests of Levels of “Type of Tests of Levels of “Type of Research Design”Research Design”

Interpretation: Small to Moderate ESs for all categories in favor of the intervention condition.

Homogeneity of ES is violated. Q-value is significant for all categories (i.e., type of research design does not explain enough variability to reach homogeneity.

Page 53: meta analysis

02/25/11 53

Sensitivity AnalysisSensitivity Analysis

• Tests the robustness of the findings• Asks the question: Will these results

stand up when potentially distorting or deceptive elements, such as outliers, are removed?

• Particularly important to examine the robustness of the effect sizes of study features, as these are usually based on smaller numbers of outcomes

Page 54: meta analysis

02/25/11 54

Selected ReferencesSelected References

Bernard, R. M., Abrami, P. C., Lou, Y. Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How Does Distance Education Compare to Classroom Instruction? A Meta-Analysis of the Empirical Literature. Review of Educational Research, 74(3), 379-439.

Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.

Hedges, L. V., Shymansky, J. A., & Woodworth, G. (1989). A practical guide to modern methods of meta-analysis. [ERIC Document Reproduction Service No. ED 309 952].