Implications for Meta-analysis Literature

Comparison of Weights in Meta-analysis Under Realistic Conditions Michael T. Brannick Liu-Qin Yang Guy Cafri

University of South Florida

Abstract Study Design

Important Notes

Study Purpose

• The overall effect size in meta-analysis is a weighted mean. Does it matter what weights we use?

Study Background— Other Weighting Schemes

• Hedges & Vevea’s (1998) approach in r

• Shrunken Estimates in r (Empirical Bayes)

• Combined Estimates in r: REVC by H&S; by H&V

• Unit Weights in r: The baseline

• Hunter & Schmidt use N; Hedges converts to z and uses N-3

• Study Background— Realistic Simulation• This simulation was based on published meta-analyses, so that values of k, N, rho ( ), and REVC ( ) would be representative of I/O meta-analyses.

We compared several weighting procedures for random-effects meta-

analysis under realistic conditions. Weighting schemes included unit,

sample size, inverse variance in r and in z, empirical Bayes, and a

combination procedure. Unit weights worked surprisingly well, and the

Hunter and Schmidt (2004) procedures worked best overall.

Results

• Published M-As• AMJ, JAP and Personnel Psychology; 1979-2005

• Inclusion criterion: effect sizes (r) available or available after conversion

• 48 M-As and 1837 effect sizes

• Inter-rater reliability: 1.0 – Ns; .99 – effect sizes (r)

• Simulation conditions formed by characteristics of published meta-analyses• Average N (N_bar) and the skewness of N distribution (N_skew) for each M-A

• A median of 168.57 for the distribution of N_bar (sampling distribution)

• A median of 2.25 for the distribution of N_skew (sampling distribution)

• Four conditions along the medians (Figure 1)

• Sampling studies for the Monte Carlo• A published M-A was randomly chosen, its K and Ns were used for that

simulation. The parameters for the simulations were chosen from:

• Choice of parameters

• The distribution of | |: 10th, 50th , and 90th percentile = .10, .22, .44, respectively

• The distribution of : 10th, 50th, and 90th percentile = .0005, .0128, and .0328

• 3 ( ) by 3 ( ) of parameter conditions• Therefore, the parameters in the simulation represent published studies

• Data generation• A Monte Carlo program written in SAS IML

• Picked an M-A under one condition of Figure 1, then picked a parameter combination

• Sampled r from a normal distribution of that and

• Meta-analyzed those sampled r(s); repeated 5000 times

• Estimators• H&S (2004) in r, H&V (1998) in z, and the other 4 approaches as described earlier

• Data analysis• and were estimated with each of 6 approaches

• Root-mean-square-difference (RMSR) between the parameter and the estimate

•Skewness in the distribution of Ns was shown to have little effect, and so simulations were rerun with only the high/low levels of N considered

•Figures 2, 4, and 6 show the empirical sampling distributions of the population mean estimates

•Figure 3, 5, and 7 show the empirical sampling distributions of the REVC estimates

The design elements had their generally expected impacts on the estimates

•The empirical sampling distributions were generally more compact with big Ns

•The means got larger when the underlying parameters increased

•The variance of the distribution increases as increases

•Provided a database and quantitative summary of published M-As of interest

•Monte Carlo simulation based on representative study characteristics

•Weights only matter when k and N are small

Conclusions

•Unit weights had surprisingly good estimates, esp. when and are large

•H&V (1998) in z performed as expected— slight overestimates

•H&S (2004) in r worked best for estimating overall mean and REVC

Study Purposes and Study Background

•Random-effect models were applied in the current study

•Sampled actual numbers of studies (K) and sample sizes (N) from the published M-As

•Used population parameters representing published M-A data

22 )1(

)1(

i

ii r

Nw

1

)1( 22

t

M N

rV

2ˆ1

MM Vw

Mi

Miii wN

rwrNEB

LN_LSLN_HSHN_LSHN_HS

1000

800

600

400

200

0Stu

dy

sa

mp

le s

izes

fro

m e

ac

h m

eta

-an

aly

sis

2ˆ

44.,22.,10. 18.,11.,02.

2

HVz1

C2C1

EB2EB1

HVz2HVr2HVr1

SH2SH1

UW2UW1

.100

0.75

0.6

0.45

0.3

0.15

0.0

Est

imat

ed R

ho

Rho=.1

.030 .017 .023 .013.028.013.043.013.024.013.027.013 RMSR

Mean of Rho estimates.099 .105.102.101.105.100

HVz1C2

C1EB2

EB1HVz2HVr2

HVr1SH2

SH1UW2

UW1

0.75

0.6

0.45

0.3

0.15

0.0

Est

imat

ed R

ho

Rho=.22

.028.037.040.055.028.034.028.033.036.036 .037.028 RMSR

Mean of Rho estimates.217 .231.236.224.231.218.219 .224.230.223.224.219

HVz1

C2C1

EB2EB1

HVz2HVr2HVr1

SH2SH1

UW2UW1

0.75

0.6

0.45

0.3

0.15

0.0

Est

imat

ed R

ho

Mean of Rho estimates

RMSR

Rho=.44

.043 .049.174.051.049.053.040 .041.159.049.041.057

.435 .458.571.464.458.437.439 .446.553.462.446.439

HVz1 C2

C1EB2

EB1HVz2HVr2

HVr1SH2

SH1UW2

UW1

0.12

0.11

0.1

0.09

0.08

0.07

0.06

0.05

0.04

0.03

0.02

0.01

0.0

Est

imat

ed t

au-s

qu

are

Tau-square= .0005

RMSR.001.004.000.016.001.004.001.03.001.017 .003.006

.011 .001.002.000.003.001.002.001.008.001.001.004

RMSR

Mean of Tau-sqaure estimates

HVz1 C2

C1EB2

EB1HVz2HVr2

HVr1SH2

SH1UW2

UW1

0.12

0.11

0.1

0.09

0.08

0.07

0.06

0.05

0.04

0.03

0.02

0.01

0.0

Est

imat

ed ta

u-sq

uare

Tau-square= .013

Mean of Tau-sqaure estimates

.016 .008 .005.006.007.007.005 .007.013.009.025.007 RMSR

.021 .015 .011.010.014.013.011 .012.012.014.020.011

HVz1 C2C1

EB2EB1

HVz2HVr2HVr1

SH2SH1

UW2UW1

0.12

0.11

0.1

0.09

0.08

0.07

0.06

0.05

0.04

0.03

0.02

0.01

0.0

Est

imat

ed ta

u-sq

uare

Tau-square= .033

Mean of Tau-sqaure estimates

RMSR.017 .014.018.057.032.013

.039

.011 .014.016.058.026.013

.033 .032.034.067.045.030 .031.032.066.042.029

Good estimator

Good Estimator

Distributions of sample sizes from published meta-analyses

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7