Schmidt & Hunter Approach to rBare Bones

Statistical Artifacts

Extraneous factors that influence observed effectSampling error*

Reliability

Range restriction

Computational error

Dichotomization of variables

*addressed in the (bare-bones) analysis

Bare Bones r

Find weighted mean and variance:

Note sample size weight.

Note that for unit weights, the weighted variance estimator is the sample, not population, estimate.

i

ii

N

rNr

i

iir N

rrNs

22 )(

Confidence Interval for Mean

i

iir N

rrNs

22 )(

k

srCI r

2

96.1%95

i

ii

N

rNr

There are k studies, with Ni observations.

This is not the only formula they use, but it’s the best one IMHO.

Estimated Sampling Error Variance

The variance of r

Nr

222 )1(

1

)ˆ1(ˆ

222

N

iei

1

)1(ˆ

222

N

re

Estimated variance for a study.

Estimated sampling variance for a meta-analysis. Note mean r is constant. This is the variance of sampling error we expect if all the studies have a common effect estimated by r-bar.

Variance of Rho

ETX 222ETX

er 222er

222er

Classical Test Theory

Sampling Error

A definition

Estimated Variance of rho

222 ˆˆ ers

2ˆ

i

ii

N

rrN 2)(

1

)1( 22

N

r-

To find the variance of infinite-sample correlations, find the variance of r in the meta-analysis and subtract expected sampling error variance. Schmidt would be quick to add that part of the estimated variance of infinite-sample correlations is artifactual.

Note that the variance of rho will be called tau-squared by Hedges

Credibility Interval

96.1%95 rCRThe credibility interval and the confidence interval are quite different things. The CI is a standard statistical estimate (intended to contain rho, or average of rho). The CR is intended to contain a percentage of the values of a random variable – infinite-sample effect sizes. The S&H value forgets that there is also uncertainty in the mean value; the two should be added. There are Bayesian programs that will do this; there is also an approximation called the prediction interval described in Borenstein et al.

Bare-Bones Example (1)Study Ni r

1 200 .20

2 100 .20

3 150 .40

4 80 .40

Mean 132.5 .30 <- Unit weighted mean

Bare-Bones Example (2)

r Ni rNi

.20 200 40

.20 100 20

.40 150 60

.40 80 32

sum 530 152

BB Example (3)

2868.530

152

i

ii

N

rNr

Recall unwighted or unit weighted mean = .30.Why are they different?

BB Example (4)r Ni

.20 200 1.507

.20 100 .753

.40 150 1.922

.40 80 1.025

Sum 530 5.208

2)( rrN i

BB Example (5)

009826.

530

208.5)( 22

i

iir N

rrNs

2868.r 5.132N

006405.15.132

)2868.1(

1

)1(ˆ

22222

N

re

003421.006405.009826.ˆˆ 222 ers

4k

384,.190.4

009826.96.12868.96.1%95

2

k

srCI r

Interpretation

Schmidt says this is a random-effects meta-analysis. It uses a sample of studies to represent a larger population of studies.

People interpret the Credibility Interval, but typically do not recognize that it is poorly estimated.