CHAPTER 4

METHODS OF COMBINING EFFECT SIZES

In this chapter we describe the standard methods of combining effect sizes from various independent studies for both point estimation as well as confidence interval estimation. We refer to Hedges and Olkin (1985) and Rosenthal (1994) for further reading.

The general principle is the following. Consider k independent studies with the ith study resulting in the estimated effect size T,, which is an estimate of the population effect size Q,, and suppose b2(T2) is the estimated variance of T,, z = 1.. . . , k . Usually, T, is based on a random sample of size n, from the ith population or study, and, in large samples, T, has an approximate normal distribution with mean Ot and variance o'(T,) = utoz,nz) . In most cases the variance ufot n,) indeed depends on Q, so that it is unknown, and 6'(T,) represents an estimate of u;oz,ntl. In some cases, T, may be stochastically independent of $(T,) (see Chapter 7).

We assume that (popularly known as the homogeneity assumption)

where Q denotes the common population effect size.

Statistical Meta-Analysis with Applications. By Joachim Hartung, Guido Knapp, Bimal K. Sinha Copyright @ 2008 John Wiley & Sons, Inc.

35

36 METHODS OF COMBINING EFFECT SIZES

Then a combined estimate of 8 is given by a weighted combination of the Ti's, namely,

(4.2)

where wi is a nonnegative weight assigned to the ith study. This very general method of linearly combining Ti's to derive an estimate of a common mean effect dates back to Cochran (1937). Clearly, for any choice of the nonstochastic weights wi's, 0 is an unbiased estimate of 8, and the weights which make Var(8) the smallest are given by

However, the above optimum weights are typically unknown since the variances o&nt) will usually be unknown and hence cannot be used. When ofot;nt, is estimated and thus replaced by $(Ti), this results in the special weighted combination

with the estimated (asymptotic) Var(8) as

(4.4)

More generally, we can also attach a quality index qi to the ith study along with the nonnegative weights wi's, thus yielding an unbiased estimate of 8 given by

with its estimated asymptotic variance as

(4.7)

In any event, when a combined estimate of 8, say T , is thus derived along with its estimated standard error given by b(T), a confidence interval for 8 with confidence level 1 - a is approximated by

LB M T - ~ , / 2 b(T) , UB z T + ~ , / 2 8(T) (4.8)

where z,12 is the upper cu/2 cut-off point obtained from a standard normal table. Moreover, if the above confidence interval does not contain 0, we reject the null hypothesis Ho : 0 = 0 at level a in favor of the alternative HI : 8 # 0. Equivalently,

METHODS OF COMBINING EFFECT SIZES 37

Study n r 1 10 0.68 2 20 0.56 3 13 0.23 4 22 0.64 5 28 0.49 6 12 -0.04 7 12 0.49 8 36 0.33 9 19 0.58 10 12 0.18

we may test the null hypothesis Ho : 0 = 0 at level a against the alternative H1 : 0 # 0 by rejecting HO if

Study n r 11 36 -0.11 12 75 0.27 13 33 0.26 14 121 0.40 15 37 0.49 16 14 0.51 17 40 0.40 18 16 0.34 19 14 0.42 20 20 0.16

Finally, based on the data from k studies, we can also test the validity of the assumption (4.1) by using an asymptotic chi-square test. Using 8, this test is based on the large- sample chi-square statistic (Cochran, 1937)

and we reject the homogeneity hypothesis HO : 191 = . . . = Bk if x2 > now discuss a few examples to illustrate the applications of the above methods.

Example 4.1. We refer to the data set from Section 18.1 dealing with validity correla- tion studies. The data are reproduced in Table 4.1 with sample size n and correlation coefficient T for each study.

We

For this data set, using ~i as Ti and recalling that

we obtain 20 m

1 - 337.002. c - = 847.185.

a = l a = 1 cT2 (T*)

and 20 ml) & = 159.687.

z = 1

38 METHODS OF COMBINING EFFECT SIZES

and

Moreover, taking Q = 0.05, we get

LB M 8 - 1 . 9 6 d G = 0.3305, UB M t? + 1 . 9 6 d a = 0.4651

For testing Ho : 0 = 0, we compute 121 = 11.58, which implies we reject H0 at level 0.05. Finally, the test for homogeneity of the Oi’s is carried out by computing x 2 = 25.65, which when compared with the table value 30.14 of x2 with 19 degrees of freedom leads to acceptance of the assumption (4.1).

Using Fisher’s 2 transformation for this data set, that is,

zi = 0.51n (:::::) - h

and recalling that Var(z,) = c, = l / ( n t - 3), we obtain

1 2‘,

20 20

5 = 201.3513. 2 ’2

c - = 530, 2 = 1 2 = 1

and 20 c = 97.4695.

i=l

and 1 -1

u2

20

c r ( ( ) FZ [ c -1 = 0.00189 i= 1

Moreover, taking Q = 0.05, we get

LB M ( - 1 . 9 G d G = 0.2948, UB RZ (+ 1 . 9 6 d e = 0.4650.

For testing H0 : C = 0, we compute 121 = 8.74, which implies we reject H0 at level 0.05. Finally, the test for homogeneity of the &’s is carried out by computing x2 = 20.97, which when compared with the table value 30.14 of x2 with 19 df leads to acceptance of the assumption (4.1). Converting results to the metric of the correlation coefficient we obtain 8 = 0.3626 with 95% confidence interval [0.2865: 0.43421.

METHODS OF COMBINING EFFECT SIZES 39

Table 4.2 Percentage of albumin in plasma protein

Variance 95% CI on mean 2 Experiment n2 Mean s,

A 12 62.3 12.986 [60.0104, 64.58961 B 15 60.3 7.840 [58.7494,61.8506] C 7 59.5 33.433 [54.1524, 64.84761 D 16 61.5 18.513 [59.2073, 63.79271

Example 4.2. Here we examine the data reported in Meier (1 953) about the percentage of albumin in plasma protein in human subjects. The data set is given in Table 4.2.

For this data set, using the mean as Ti and the variance of Ti as &‘(Ti) = s?/ni, we obtain

and

This leads to

and

4 - 4 - 1

= 238.5492, c - 6z(Ti) = 3.9110. z = 1 i=l

4 & = 14553.47. a = 1

Moreover, taking Q: = 0.05, we get

LB % e - 1 . 9 6 m = 60.0038, UB FZ 6 + 1 . 9 6 m = 61.9860

Finally, the test for homogeneity of the 8,’s is carried out by computing x2 = 3.1862, which when compared with the table value 7.815 of x2 with 3 df leads to acceptance of the assumption (4.1).

Example 4.3. These data are quoted from Eberhardt, Reeve, and Spiegelman (1989) and deal with the problem of estimation of mean selenium in nonfat milk powder by combining the results of four methods. The data are given in Table 4.3.

For this data set, using the mean as T, and the variance as 6’(T,) = s;/n,, we obtain

4 - 4 - 1

- 661.9528. C - - - 6.0396. t=l 2 = 1 d2(Ta)

40 METHODS OF COMBINING EFFECT SIZES

Table 4.3 Selenium in nonfat milk powder

Variance 95% CI on mean 2 Methods n2 Mean s2

Atomic absorption spectrometry 8 105.0 85.711 [ 97.2601, 112.73991 Neutron activation: 1. Instrumental 12 109.75 20.748 [106.8559, 112.64411 2. Radiochemical 14 109.5 2.729 [108.5462, 110.45381 Isotope dilution mass spectrometry 8 113.25 33.640 [108.4011, 118.09891

and 4

= 72556.6. i=l

and 1 4

%(r(e) x [ -]-I = 0.1656. i=l 0 2 (Ti)

Moreover, taking cy = 0.05, we get

LB x 6 - 1 . 9 6 m = 108.8045, UB M 6 + 1 . 9 6 m = 110.3996.

Finally, the test for homogeneity of the Qi’s is carried out by computing x 2 = 5.2076, which when compared with the table value 7.815 of x 2 with 3 df leads to acceptance of the assumption (4.1).

Example 4.4. The results of nine randomized controlled trials comparing SMFP to NaF dentifrices (toothpastes) in the prevention of caries development have been reanalyzed in Abrams and Sanso (1998). Let us consider the difference of means as the parameter of interest. Then, Table 4.4 contains the observed differences of means and corresponding standard errors for the nine studies. More details on this data set can be found in Section 18.3.

For this data set, using the difference of means as Ti and the variance as the squared standard error, we obtain

1 Ti - 33.2037, = 117.2006, -

9 9

6-2(Ti) i=l i=l

and 9

= 14.7873. i= 1

METHODS OF COMBINING EFFECT SIZES 41

Table 4.4 Nine randomized trials comparing SMFP to NaF dentifrices in the prevention of caries development

Difference Study SMFP - NaF

Standard error

1 0.86 0.5756 2 0.33 0.5610 3 0.47 0.3507 4 0.50 0.25 11 5 -0.28 0.5404 6 0.04 0.275 1 7 0.80 0.7826 8 0.19 0.1228 9 0.49 0.2784

and 1 9

G(8) = [ -1 -’ = 0.0085. i=l .‘(Ti)

Moreover, taking Q = 0.05, we get

LB = 8 - 1.964- = 0.1022, UB = 8 + l . 9 6 4 G = 0.4644.

Finally, the test for homogeneity of the 8i’s is carried out by computing x2 = 5.38, which when compared with the table value 15.51 of x2 with 8 df leads to acceptance of the assumption (4.1).