curran,west&finch (1996)

7/25/2019 Curran,West&Finch (1996)

1/14

Psycholog ica l Met hods Copy righ t 1996 by the Am erican Psycholog ica l Assoc ia t io n , Inc .

1996. VoL 1, No. l, 16-29 1082-989X/96/ 3.00

T h e Rob u s tn es s o f T es t S ta t i s t i c s to Non n orm al i ty

and Spec i f icat ion Error in Co nf irmatory Factor nalys i s

Patrick J Curran

Uni ve r s i t y o f Ca l i f o r n ia , L os A nge l es

Stephen G West

Ar i zona S t a t e Uni ve r s i t y

John F Finch

T e x a s A M U n i v e r si t y

M o n t e C a r l o c o m p u t e r s i m u l at io n s w e r e u s e d t o i n v e s ti g a te t h e p e r f o r m a n c e

o f t h r e e X2 es t s t a ti s ti c s in con f i r ma t or y f ac t o r ana lys i s ( CF A) . No r mal t h eor y

m a x i m u m l i k e l i h o o d )~2 ( M L ) , B r owne ' s a sympt o t i c d i s t r i bu t i on f r ee X2

( ADF ) , and t he S a t o r r a - Ben t l e r r e sca l ed X2 ( S B) wer e exam i ned un der va r y-

ing condi t ions of sample s ize , model speci f icat ion, and mul t ivar ia te di s t r ibu-

t ion . F or p r ope r l y spec i f ied mode l s , M L a nd S B showed no ev i dence o f b ia s

unde r nor m a l d i s t r ibu t i ons ac r os s a ll s ampl e s izes, wher eas A DF was b i a sed

a t a l l bu t t he l a r ges t s amp l e s izes. M L was i nc r eas i ng ly ove r es t i ma t ed w i t h

i nc r eas i ng nonnor mal i t y , bu t bo t h S B ( a t a l l s ampl e s i zes ) and ADF ( on l y

a t l a r ge s amp l e s izes ) showed no ev i dence o f b i a s . F or mi s spec i f ied mode l s ,

M L was aga i n i n f l a t ed w i t h i nc r eas i ng nonnor mal i t y , bu t bo t h S B and ADF

wer e un de r es t i ma t ed w i t h inc r eas ing nonno r mal i t y . I t app ea r s t ha t t he power

of t he S B and A D F t e s t s ta t i st i c s t o de t ec t a mo de l m i s speci f ica t ion i s a t t enu-

a t ed g i ven nonnor mal l y d i s t r i bu t ed da t a .

C o n f i r m a t o r y f a c to r a n al y si s ( C F A ) h a s b e c o m e

a n i n c r e a si n g ly p o p u l a r m e t h o d o f in v e s t ig a t in g

t h e s t r u c t u r e o f d a t a s e t s i n p s y c h o l o g y . I n c o n t r a s t

t o t r a d it i o n a l e x p l o r a t o r y f a c t o r a n a ly s i s t h a t d o e s

n o t p l a c e s t r o n g a p r i o r i r e s t r i c t i o n s o n t h e s t r u c -

t u r e o f t h e m o d e l b e i n g t e s te d , C F A r e q u i r e s t h e

i n v e s t i g a to r t o s p e c i fy b o t h t h e n u m b e r o f fa c t o r s

P a t r i ck J . Cur r an , Gr adua t e S choo l o f E duca t i on ,

Uni ve r s i t y o f Ca l i fo r n i a , L os Ange l es ; S t ephen G . W es t ,

De par t m ent o f P sycho l ogy , Ar i zon a S t a t e Uni ve r s i ty ;

John F . Fi nch , De par t m ent o f P sycho l ogy , T exas

A M Uni ve r s i t y .

W e t h a n k L e o n a A i k e n , P e t e r B e n t l e r, D a v i d K a p l a n ,

D a v i d M a c K i n n o n , B e n g t M u t h 6 n , M a r k R e i s e r , a n d

Al be r t S a t o r r a f o r t he i r va l uab l e i npu t . T h i s wor k was

pa r t i a l l y suppor t ed by Na t i ona l I ns t i t u t e on A l coho l

Abuse and Al coho l i sm P os t doc t o r a l F e l l owsh i p F 32

AA05402-01 and G r an t P 50M H 39246 dur i ng t he wr it i ng

of thi s ar t ic le .

Cor r esp onde nce co nce r n i ng t h i s a r t ic l e shou l d be ad-

dr es sed t o P a t r i ck J. Cur r an , G r adu a t e S choo l o f E duca -

t ion , U ni ve r s i ty o f Ca l i f o r n i a, L os A nge l es , 2318 M oor e

Hal l , Los Angeles , Cal i fornia 90095-1521. E lect ronic

mai l may be s en t v i a I n t e r ne t t o i dhbp j c@mvs .

oac.ucla .edu.

a n d t h e s p e c i f ic p a t t e r n o f l o a d i n g s o f e a c h o f t h e

m e a s u r e d v a r i a b l e s o n t h e u n d e r l y i n g s e t o f f a c-

t o rs . I n ty p i c a l s i m p l e C F A m o d e l s , e a c h m e a s u r e d

v a r i a b l e i s h y p o t h e s i z e d t o l o a d o n o n l y o n e f a c to r ,

a n d p o s i ti v e , n e g a ti v e , o r z e r o ( o r t h o g o n a l ) c o r r e -

l a t i o n s a r e s p e c i fi e d b e t w e e n t h e f a c t o r s . S u c h

m o d e l s c a n p r o v i d e s tr o n g e v i d e n c e a b o u t t h e c o n -

v e r g e n t a n d d i s c r im i n a n t v a l i d i ty o f a s e t o f m e a -

s u r e d v a r i a b l e s a n d a l l o w t e s t s a m o n g a s e t o f

t h e o r i e s o f m e a s u r e m e n t s t r u c tu r e . M o r e c o m p l i -

c a t e d C F A m o d e l s m a y s p e c i fy m o r e c o m p l e x p a t-

t e r n s o f f a c t o r lo a d i n gs , c o r r e l a t i o n s a m o n g e r r o r s

o r s p e c i f ic fa c t o r s , o r b o t h . I n a l l c a s e s , C F A m o d -

e l s s e t r e s t ri c t i o n s o n t h e f a c t o r l o a d i n g s , t h e c o r r e -

l a t i o n s b e t w e e n f a c t o r s , a n d t h e c o r r e l a t i o n s b e -

t w e e n e r r o r s o f m e a s u r e m e n t t h a t p e r m i t t e s ts o f

t h e f it o f th e h y p o t h e s i z e d m o d e l t o t h e d a t a .

T h e r e a r e t w o g e n e r a l c l a ss e s o f a s s u m p t i o n s

t h a t u n d e r l i e t h e s t a t i s t i c a l m e t h o d s u s e d t o e s t i -

m a t e C F A m o d e l s : d i s t r i b u t i o n a l a n d s t r u c t u r a l

( S a t o r r a , 1 9 9 0 ) . N o r m a l t h e o r y m a x i m u m l i k e l i -

h o o d ( M L ) e s t i m a t i o n h a s b e e n u s e d t o a n a l y z e

t h e m a j o r it y o f C F A m o d e l s . M L m a k e s t h e dis tr i

b u t i o n a l a s s u m p t i o n t h a t t h e m e a s u r e d v a r i a b l e s

h a v e a m u l t i v a r i a te n o r m a l d i s t ri b u t io n i n t h e p o p -

16


2/14

CHI-SQUARE TEST STATISTICS 17

ulation. However, the majority of data collected

in behavioral research do not follow univariate

normal distributions, let alone a multivariate nor-

mal distribution (Micceri, 1989). Indeed, in some

important areas of research such as drug use, child

abuse, and psychopathology, it would not be rea-

sonable to even expect that the observed data

would follow a normal distribution in the popula-

tion. In addition to the distributional assumption,

ML (and all methods of estimation) makes the

structur l

assumption that the structure tested in

the sample accurately reflects the structure that

exists in the population. If the sample structure

does not adequately conform to the corresponding

population structure, severe distortions in all as-

pects of the final solution can result.

Although the chi-square test statistic can be

used to measure the extent of the violation of the

structural assumpt ion (Lawley Maxwell, 1971),

the accuracy of this test statistic can be compro-

mised given violation of the distributional assump-

tion (Satorra, 1990). Violations of both the distri-

butional and structural assumptions are common

(and often unavoidable) in practice and can poten-

tially lead to seriously misleading results. It is thus

important to fully understand the effects of the

multivariate nonnormality and specification error

on maximum likelihood estimation and other al-

ternative estimators used in CFA.

Methods of Estimation

By far the most common method used to esti-

mate confirmatory factor models is normal theory

ML. Nearly all of the major software packages use

ML as the standard default estimator (e.g., EQS,

Bentler, 1989; LISREL, Jtireskog S6rbom,

1993; PROC CALLS, SAS Institute, Inc., 1990;

RAMONA, Browne, Mels, Coward, 1994).

Under the assumptions of multivariate normality,

proper specification of the model, and a suffi-

ciently large sample size (N), ML provides asymp-

totically (large sample) unbiased, consistent, and

efficient parameter estimates and standard errors

(Bollen, 1989). An important advantage of ML is

that it allows for a formal statistical test of model

fit. (N - 1) multiplied by the minimum of the ML

fit function is distributed as a large sample chi-

square with 1/2(p)(p + 1) -t degrees of freedom,

where p is the number o f observed variables and

t is the number of freely estimated parameters

(Bollen, 1989).

One potential limitation of ML estimation is

the strong assumption of multivariate normality.

Given the presence of non-zero third- and (partic-

ularly) fourth-order moment s (skewness and kur-

tosis, respectively),1 the resulting ML parameter

estimates are consistent but no t efficient, and the

minimum of the ML fit function is no longer dis-

tributed as a large sample central chi-square. In-

stead, (N - 1) multiplied by the minimum of the

ML fit function generally produces an inflated

(positively biased) estimate of the referenced chi-

square distribution (Browne, 1982; Satorra, 1991).

Hence, using the normal theory chi-square statistic

as a measure of model fit under conditions of non-

normality will lead to an inflated Type I er ror rate

for model rejection. Consequently, in practice a

researcher may mistakenly reject or opportunisti-

cally modify a model because the distribution of

the observed variables is not multivariate normal

rather than because the model itself is not correct

(see MacCallum, 1986; MacCallum, Roznowski,

Necowitz, 1990).

Several different approaches have been pro-

posed to address the problems with ML estimation

under conditions of multivariate nonnormality.

One example is the development of alternative

methods of estimation that do not assume multi-

variate normality. One such estimator that is cur-

rently available in structural modeling programs

such as EQS (Bentler, 1989), LISCOMP (Muthrn,

1987), LISRE L (Jr reskog SOrbom, 1993), and

RAMONA (Browne et al., 1994) is Browne's

(1982, 1984) asymptotic distribution free (ADF)

method of estimation. The derivation of the ADF

estimator was not based on the assumption of mul-

tivariate normality so that variables possessing

non-zero kurtoses theoretically pose no special

problems for estimation. ADF provides asymptot-

ically consistent and efficient parameter estimates

and standard errors, and (N - 1) times the mini-

mum of the fit function is distributed as a large

sample chi-square (Browne, 1984). One practical

disadvantage of ADF is that it is computationally

1The multivariate normal distribution is actually char-

acterized by skewness equal to 0 and kurtosis equal to 3.

However, it is common practice to subtract the constant

value of 3 from the kurtosis estimate so that the normal

distribution is characterized by zero skewness and zero

kurtosis. We will similarly refer to the normal distribu-

tion as defined by zero skewness and zero kurtosis.


3/14

18 CURRAN, WEST, AND FINCH

very demanding. Browne (1984) anticipated that

models with greater than 20 variables could not

be feasibly estimated with ADF. A second possible

disadvantage is that initial findings suggest that

this estimator performs poorly at the small to mod-

erate sample sizes that typify much of psychologi-

cal research.

A second approach that has been developed

for computing a more accurate test statistic under

conditions of nonnormal ity is to adjust the normal

theory ML chi-square estimate for the presence

of non-zero kurtosis. Because the normal theory

chi-square does not follow the expected chi-square

distribution under conditions of nonnormality, the

normal theory chi-square must be corrected, or

rescaled, to provide a statistic that more closely

approximates the referenced chi-square distribu-

tion (Browne, 1982, 1984). One variant of the re-

scaled test statistic that is currently only available

in EQS (Bentler , 1989) is the Satorra-Bent ler chi-

square (SB X2; Satorra, 1990, 1991; Satorra

Bentler, 1988). The SB X2 corrects the normal the-

ory chi-square by a constant k, a scalar value that

is a function of the model implied residual weight

matrix, the observed multivariate kurtosis, and the

model degrees of freedom. The greater the degree

of observed multivariate kurtosis, the greater

downward adjustment that is made to the inflated

normal theory chi-square.

Review of Monte Carlo Studies

Muth6n and Kaplan (1985) studied a properly

specified four indicator single-factor model under

five distributions ranging from normal to severely

nonnormal for one sample size (1,000). For univar-

iate skewness greater than 2.0, the ML X2 was

clearly inflated whereas the ADF X2remained con-

sistent. Muthrn and Kaplan (1992) extended these

findings by adding more complex model specifica-

tions, an addi tional sample size (500), and increas-

ing the number of replications to 1,000 per condi-

tion. The normal theory chi-square was extremely

sensitive to both nonnormality and model com-

plexity (defined as the number o f parameters esti-

mated in the model). The ADF X2 appeared to be

very sensitive to model complexity, with extreme

inflation of the model chi-square as the tested

model became increasingly complex. The ADF X2

was also particularly inflated at the smaller sam-

ple size.

Satorra and Bent ler (1988) per formed a Monte

Carlo simulation using a proper ly specified four in-

dicator single-factor model to evaluate the behav-

ior of the SB X2 est statistic. The unique variances

of the four indicators were calculated with univari-

ate skewness of 0 and a homogenous univariate kur-

tosis of 3.7. The models were estimated using ML,

unweighted least squares (ULS), and ADF, based

on 1,000 replications of a single sample size of 300.

The normal theory ML X2and the SB X2performed

similarly to one another. On average, the ML X2

slightly underestimated the expected value of the

model chi-square while the SB g 2 slightly overesti-

mated the expected value. However, the ML

X z

had

a larger variance than did the SB X2. The ADF X2

resulted in the highest average value, although it

also attained the lowest variance.

Chou, Bentler , and Sa torra (1991) similarly used

a Monte Carlo simulation to examine the ML,

ADF, and SB X2 est statistics for a properly speci-

fied model under varying conditions of normality.

A two-factor six indicator CFA model was repli-

cated 100 times per condi tion based on two sample

sizes (200 and 400) and six multivariate distribu-

tions. Two versions of the model were estimated,

one in which all of the necessary parameters were

freely estimated, and one in which the factor load-

ings were fixed to the population values. Consis-

tent with previous research, the ML X2was inflated

under nonnormal conditions. The SB

X

outper-

formed both the ML and ADF X2 test statistics in

nearly all conditions.

Finally, Hu, Bentler, and Kano (1992) performed

a major simulation study based on a three-factor

confirmatory factor model with five indicators per

factor. Six sample sizes were used (ranging from 150

to 5,000) with 200 replications per condition. Seven

different symmetric distributions were considered,

ranging from normal to severely nonnormal (high

kurtosis). The normal theory estimators (maximum

likelihood and generalized least squares) provided

inflated chi-square values as nonnormality in-

creased. The ADF test statistic was relatively unaf-

fected by distribution but was only reliable at the

largest sample size (5,000). Finally, the SB X2 per-

formed the best of all test statistics, although mod-

els were rejected at a higher frequency than was ex-

pected at small sample sizes.

In summary, Monte Carlo simulation studies

have consistently supported the theoretical predic-

tion that the normal theory ML X2 test statistic is


4/14

C H I - S Q U A R E T E S T S T A T I S T I C S 1 9

s i g n if i c a n tl y i n f l a te d a s a f u n c t i o n o f m u l t i v a r i a t e

n o n n o r m a l i t y . T h e A D F X 2 is t h e o r e t i c a ll y a s y m p -

t o ti c a ll y r o b u s t t o m u l t i v a r i a te n o n n o r m a l i t y , b u t

i ts b e h a v i o r a t s m a l l e r ( a n d m o r e r e a l is t i c ) s a m p l e

s i ze s i s p o o r . T h e S B X 2 h a s n o t b e e n f u l l y e x a m -

i n e d , b u t i n i t i a l f i n d i n g s i n d i c a t e t h a t i t o u t p e r -

f o r m s b o t h t h e M L a n d A D F t e s t s ta t is ti c s u n d e r

n o n n o r m a l d i s t r ib u t i o n s, a l t h o u g h i t d o e s t e n d t o

o v e r r e j e c t m o d e l s a t s m a l l e r s a m p l e s iz e s.

A l t h o u g h t h e r a m i f i c a t i o n s o f v io l a t i n g t h e d i s tr i -

b u t i o n a l a s s u m p t i o n s i n C F A i s b e c o m i n g b e t t e r

u n d e r s t o o d , m u c h l e ss is k n o w n a b o u t v i o l at i o n s o f

t h e s t r u c t u r a l a s s u m p t i o n . R e c e n t w o r k h a s a d -

d r e s s e d t h e e f f e c t s o f m o d e l m i s s p e c i f i c a ti o n o n t h e

c o m p u t a t i o n o f p a r a m e t e r e s t i m a t e s a n d st a n d a r d

e r r o r s ( K a p l a n , 1 9 8 8 , 19 8 9 ) a s w e l l a s p o s t h o c

m o d e l m o d i f i c a t i o n ( M a c C a l l u m , 1 9 8 6 ) a n d t h e

n e e d f o r a l t e r n a t i v e i n d i c e s o f f it ( M a c C a l l u m ,

1 9 90 ). H o w e v e r , l i tt le is k n o w n a b o u t t h e b e h a v i o r

o f c h i - s q u a r e t e s t s ta t is t ic s u n d e r s i m u l t a n e o u s v i o -

l a t i o n s o f b o t h t h e d i s t r i b u t i o n a l a n d t h e s t r u c t u r a l

a s s u m p t i o n s . I n d e e d , w e a r e n o t a w a r e o f a s i n g le

e m p i r i c al s tu d y t h a t h a s e x a m i n e d t h e A D F a n d S B

A 2 t e s t s t a t is t ic s u n d e r t h e s e t w o c o n d i t i o n s . G i v e n

t h e l o w p r o b a b i l i t y t h a t t h e s t r u c t u r e t e s t e d i n a

s a m p l e p r e c i s e l y c o n f o r m s t o t h e s t r u c t u r e t h a t e x -

i s ts in t h e p o p u l a t i o n , i t is c r i t i c a l t h a t a b e t t e r u n -

d e r s t a n d i n g b e g a i n e d o f t h e b e h a v i o r o f t h e t e s t

s t at i st i cs u n d e r t h e s e m o r e r e a l is t i c c o n d i t io n s .

T h e P r e s e n t S t u d y

A s e ri e s o f M o n t e C a r l o c o m p u t e r s i m u l a t i on s

w e r e u s e d t o s t u d y t h e e f f e c t s o f s a m p l e s i z e , m u l t i -

v a r i a t e n o n n o r m a l i t y , a n d m o d e l s p e c i fi c a ti o n o n

t h e c o m p u t a t i o n o f t h r e e c h i - s q u a r e t e s t s t at i st i cs

t h a t a r e c u r r e n t l y w i d e l y a v a i l a b l e t o t h e p r a c t i c i n g

r e s e a r c h e r : M L , S B , a n d A D F . 2 F o u r s p e c i f i c a t io n s

o f a n o b l i q u e t h r e e - f a c t o r m o d e l w i t h t h r e e i n d ic a -

t o r s p e r f a c t o r w e r e c o n s i d e r e d . T h e f ir st t w o m o d -

e l s w e r e

correc t l y spec i f i ed

s u c h t h a t t h e s t r u c t u r e

e s t i m a t e d i n t h e s a m p l e p r e c i s e l y c o r r e s p o n d e d t o

t h e s t r u c t u r e t h a t e x i s t e d i n t h e p o p u l a t i o n . T h e

s e c o n d t w o m o d e l s w e r e m i s s p e c i f i e d s u c h t h a t t h e

s t r u c t u r e t e s t e d i n t h e s a m p l e d i d

n o t

c o r r e s p o n d

t o t h e s t r u c t u r e t h a t e x i s t e d i n t h e p o p u l a t i o n .

M e t h o d

M o d e l S p e c i f i c a t i o n

F o u r s p e c if i ca t io n s o f a n o b l i q u e t h r e e - f a c t o r

m o d e l w i t h t h r e e i n d i c a t o r s p e r f a c t o r w e r e e x a m -

i n e d. T h e b a s ic c o n f i r m a t o r y f a c t o r m o d e l i s p r e -

s e n t e d i n F i g u r e 1 . T h e p o p u l a t i o n p a r a m e t e r s

c o n s i s t e d o f f a c t o r l o a d i n g s ( e a c h A = . 70 ) , u n i q u e -

n e s s e s ( e a c h O ~ = . 51 ) , i n t e r f a c t o r c o r r e l a t i o n s

( e a c h ~b = . 3 0 ), a n d f a c t o r v a r i a n c e s ( a l l s e t t o 1 . 0 ) .

M o d e l 1 M o d e l 1 w a s p r o p e r l y s p e c i f i e d s u c h

t h a t t h e m o d e l t h a t w a s e s t i m a t e d i n t h e s a m p l e

d i r e c tl y c o r r e s p o n d e d t o t h e m o d e l t h a t e x i s t e d in

t h e p o p u l a t i o n . T h u s , b o t h t h e s a m p l e a n d t h e

p o p u l a t i o n m o d e l s c o r r e s p o n d e d t o t h e s o l id l in e s

p r e s e n t e d i n F i g u r e 1 .

M o d e l 2

M o d e l 2 c o n t a i n e d t w o f a c t o r lo a d -

i n g s t h a t w e r e e s t i m a t e d i n t h e s a m p l e b u t d i d

n o t

e x i s t i n t h e p o p u l a t i o n . T h u s , i n F i g u r e 1 , t h e

d o u b l e d a s h e d l in e s r e p r e s e n t t h e t w o f a c t o r lo a d -

i n g s t h a t l i n k e d I t e m 5 t o F a c t o r 3 a n d I t e m 8 t o

F a c t o r 2 , an d t h e e x p e c t e d v a l u e o f t h e s e p a r a m e -

t e r s w a s 0 . T h i s i s a m i s s p e c i f i c a t i o n o f i nc lus ion

N o t e t h a t f r o m t h e s t a n d p o i n t o f st a t is t ic a l th e o r y ,

e s t i m a t io n o f p a r a m e t e r s w i th a n e x p e c t e d v a l u e

o f 0 i n th e p o p u l a t i o n d o e s n o t b i a s t h e s a m p l e

r e s u lt s . M o d e l 2 is t h u s c o n s i d e r e d t o b e a p r o p e r l y

s p e c i f ie d m o d e l .

M o d e l 3

M o d e l 3 e x c l u d e d t w o l o a d in g s f r o m

t h e s a m p l e t h a t

d i d

e x i s t i n t h e p o p u l a t i o n . T h u s ,

i n F i g u r e 1 , t h e s i n g l e d a s h e d l i n e s r e p r e s e n t t h e

t w o e x c l u d e d f a c t o r l o a d i n g s ( b o t h p o p u l a t i o n

A s = . 35 ) t h a t l i n k e d I t e m 6 t o F a c t o r 3 a n d I t e m

7 t o F a c t o r 2 . T h e v a l u e o f A = .3 5 w a s c h o s e n t o

r e f l e c t a s m a ll t o m o d e r a t e f a c t o r l o a d i n g t h a t

m i g h t b e c o m m o n l y e n c o u n t e r e d i n p ra c t ic e . T h is

i s a m i s s p e c i f i c a t i o n o f

exc lus ion

M o d e l 4

F i n a l l y , M o d e l 4 w a s t h e c o m b i n a -

t i o n o f M o d e l s 2 a n d 3 . L i k e M o d e l 2 , tw o f a c t o r

l o a d i n g s w e r e e s t i m a t e d i n t h e s a m p l e t h a t d i d

n o t

e x i s t i n t h e p o p u l a t i o n ( t h e d o u b l e d a s h e d

l i n e s l i n k i n g I t e m 5 t o F a c t o r 3 a n d I t e m 8 t o

F a c t o r 2 ) . A d d i t i o n a l l y , l i k e M o d e l 3 , t w o f a c t o r

l o a d in g s w e r e e x c l u d e d f r o m t h e s a m p l e t h a t

d i d

e x i s t i n t h e p o p u l a t i o n ( t h e s i n g le d a s h e d l i n e s

t h a t l i n k e d I t e m 6 t o F a c t o r 3 a n d I t e m 7 t o

F a c t o r 2 ) . T h i s i s a m i s s p e c if i c a t io n o f b o t h

i nc lus ion

a n d

exc lus ion

2 Note tha t G LS i s a lso ava i l ab le in s t anda rd packages

and is re la t ively widely used. H ow ever , GL S is a norm al

theory e s t im a tor tha t i s a sym pto t i ca l ly equ iva len t to

ML , and p reviou s s tudies (e .g. , Mu th6n & Kaplan , 1985,

1992) have shown the behav ior o f ML and GLS to be

ve ry similar.


5/14

2 0

C U R R A N W E S T A N D F I N C H

. 5 1 . 5 1 . 5 1 . 5 1 . 5 1 . 5 1 . 5 1 . 5 1 . 5 1

. . 7 0 \ . 70 1 . 7 0 / ' ~ . - ' ' ' ~ / ' \ 7 0 1 7 0

1 70

a o a o

. 3 0

Figure 1 Nine - indicator three- facto r obl ique co nf irmato ry factor analys is m odel with

populat ion parameter values . Solid l ines represent parameters that were shared be-

tween the sample and the populat ion; s ingle dashed l ines represent parameters that

ex i s ted in the popula t ion ) t = . 35) bu t wer e om i t ted f rom the s am ple ; dou ble dashed

lines repres ent param eters that did not exis t in the popu lat ion ) t = 0) but were

es t imated in the sample .

Cond i t ions

Multivariate distr ibutions

T h r e e p o p u l a t i o n

d i s t r i b u t i o n s w e r e c o n s i d e r e d f o r a l l f o u r m o d e l

s p e c i fi c a t io n s s e e F i g u r e 2 ). D i s t r i b u t i o n 1 w a s

m u l t i v a r i a t e n o r m a l w i t h u n i v a r i a t e s k e w n e s s a n d

k u r t o s e s e q u a l t o 0 . D i s t r i b u t i o n 2 w a s m o d e r a t e l y

n o n n o r m a l w i t h u n i v a r ia t e s k e w n e s s o f 2 .0 a n d

k u r t o s e s o f 7 .0 . F i n a ll y , D i s t r i b u t i o n 3 w a s s e v e r e l y

n o n n o r m a l w i t h u n i v a r i a t e s k e w n e s s o f 3 . 0 a n d

k u r t o s e s o f 2 1 . 0 . T h e s e l e v e l s o f n o n n o r m a l i t y

w e r e c h o s e n t o r e p r e s e n t m o d e r a t e a n d s e v e r e

n o n n o r m a l i t y b a s e d o n o u r e x a m i n a t i o n o f t h e

l e v e ls o f s k e w n e s s a n d k u r t o s e s i n d a t a s e ts f r o m

s e v e ra l c o m m u n i t y - b a s e d m e n t a l h e a l t h a n d s u b -

s t a n c e a b u s e s t u d i e s .

Sam ple s i z e F o u r s a m p l e s i z e s w e r e c o n s i d -

e r e d f o r a l l m o d e l s p e c i f i c a t i o n s : 1 0 0 , 2 0 0 , 5 0 0 ,

and 1 , 000 .

Replications A l l m o d e l s w e r e r e p l i c a t e d 2 0 0

t i m e s p e r c o n d i t i o n .

Data generation T h e r a w d a t a w e r e g e n e r a t e d

u s in g b o t h t h e P C a n d m a i n f r a m e v e r s io n o f E Q S

V e r s i o n 3 ; B e n t l e r , 1 9 8 9 ) . D e t a i l s o f t h e d a t a g e n -

e r a t i o n p r o c e d u r e a r e p r e s e n t e d i n t h e A p p e n d i x .

M e a s u r e s

T h r e e c h i - s q u a r e t e s t s t a t i s t i c s w e r e s t u d i e d :

n o r m a l t h e o r y M L , A D F , a n d t h e S B s c al e d X2.

A l l t h r e e t e s t s t a t i s t i c s w e r e c o m p u t e d b y E Q S

V e r s i o n 3 . 0 ). N o t e t h a t t h e M L a n d A D F s t at is t ic s

p r o v i d e d b y E Q S s h o u l d b e i d e n t i c a l t o t h a t

a v a i la b l e t h r o u g h c u r r e n t v e r s io n s o f L I S C O M P ,

L I S R E L , a nd R A M O N A .

R e s u l t s

Expec ted Va lue o f T es t S ta t i s t i c s

T h e e x p e c t e d v a l u e s o f t h e c h i - s q u a r e t e s t s ta t is -

t i c s f o r M o d e l s 1 a n d 2 w e r e s i m p l y t h e m o d e l

d e g r e e s o f f r e e d o m f o r al l t h r e e e s t i m a t o r s a c r o s s

a ll d i s t r ib u t i o n s a n d s a m p l e s i z es 2 4 .0 fo r M o d e l

1 a n d 2 2 . 0 f o r M o d e l 2 ) . B e c a u s e M o d e l s 3 a n d 4

w e r e m i s s p e c if i e d , t h e e x p e c t e d v a l u e s o f t h e s e

t e s t s t a t i s t i c s c o u l d n o t b e c o m p u t e d d i r e c t l y . I n -

s t ea d , t h e e x p e c t e d v a l u es w e r e c o m p u t e d a s la r ge

s a m p l e e m p i r i c a l e s t i m a t e s t h a t d i f f e r e d a s a f u n c -

t i o n o f m e t h o d o f e s t im a t i o n , m u l t i v a r i a t e d i s tr i-

b u t i o n , a n d s a m p l e s i z e. F u r t h e r d e t a i ls r e g a r d i n g

t h e c o m p u t a t i o n s o f t h e se e s t i m a t e s a r e p r e s e n t e d

i n th e A p p e n d i x .

M o n t e C a r l o R e s u l t s

T a b l e s 1 , 2 , 3 , a n d 4 p r e s e n t t h e m e a n o b s e r v e d

v a l u e , t h e e x p e c t e d v a l u e , t h e p e r c e n t a g e o f b i a s,


6/14


~J

t -

Q

O

U .

5 [ ] 0 0 -

4 0 0 0

3 0 0 0

200O

1000

:

- 4 . 5 - 3 5

- 2 . 5 - 1 5

f ~

I I

I I

I I

I I

I . , |

, I

\

4 , 5 0 . 5

1 . 5 2 . 5 3 . 5 4 . 5

S t a n d a r d i z e d V a l u e

Figure 2 . P lo t s o f norm a l ( so lid l ine) , m od era te ly non-

norm al (do t t ed l ine ) , and s eve re ly nonnorm al (dashed

l ine ) em pi rica l d is t r ibu t ions based on a rand om sam ple

of N = 10,000.

a n d t h e p e r c e n t a g e o f m o d e l s r e j e c t e d a t p < .0 5

f o r t h e M L , S B , a n d A D F X 2 t e s t s t at i st i cs f o r

a l l f o u r m o d e l s p e c i f i c a t i o n s ? F o r t h e c o r r e c t l y

s p e c i f ie d m o d e l s ( M o d e l s 1 a n d 2 ) , t h e e x p e c t e d

r e j e c t i o n r a t e w a s 5 ; a n d g i v e n 2 0 0 r e p l i c a t i o n s

a n d a = . 05 , t h e 9 5 c o n f i d e n c e i n t e r v a l f o r t h e

p e r c e n t a g e o f r e j e c t e d m o d e l s d e f i n e d a n a p p r o x i -

m a t e u p p e r a n d l o w e r b o u n d o f 2 a n d 8 , r e s p e c-

t i v el y . R e j e c t i o n r a t e s f o r t h e o b t a i n e d c h i - s q u a r e

v a l u e s f a l li n g w i t h i n t h e s e b o u n d s a r e c o n s i s t e n t

w i t h t h e n u l l h y p o t h e s i s t h a t t h e e s t i m a t o r i s u n b i -

a s ed . M o d e l r e j e c t io n r a t e s w e r e n o t a s m e a n i n g f u l

f o r m i s s p e c i f ie d m o d e l s ( M o d e l s 3 a n d 4 ) , s o r e l a -

t i ve b ia s w a s c o m p u t e d ( t h e o b s e r v e d v a l u e m i n u s

t h e e x p e c t e d v a l u e d i v i d e d b y t h e e x p e c t e d v a l u e ) .

B i a s i n e x c e s s o f 1 0 w a s c o n s i d e r e d s i g n if i c a n t

( K a p l a n , 1 9 8 9 ) .

M o d e l S p e c i f i c a t i o n 1 .

T a b l e 1 p r e s e n t s t h e r e -

s u l ts f o r M o d e l 1 . R e c a l l t h a t M o d e l 1 w a s p r o p e r l y

s p e c i f ie d s u c h t h a t t h e m o d e l e s t i m a t e d i n t h e s a m -

p l e d i re c t l y c o r r e s p o n d e d t o t h e m o d e l t h a t e x i s t e d

i n th e p o p u l a t i o n . T h e e x p e c t e d v a l u e f o r a l l t h r e e

te s t s t a t i s t i c s wa s E (X 2) = 24 . 0 .

U n d e r m u l t i v a r ia t e n o r m a l i t y , t h e M L X 2 r e -

j e c t e d t h e e x p e c t e d n u m b e r o f m o d e l s a c r o ss a ll

s a m p l e s iz e s ( a p p r o x i m a t e l y 5 ) . C o n s i s t e n t w i t h

b o t h t h e o r y a n d p r e v i o u s s i m u l a t io n r e s e a r c h , t h e

M L X 2 b e c a m e i n c r e a s i n g l y p o s i t iv e l y b i a s e d a s

t h e d i s tr i b u ti o n b e c a m e i n c r e a s i ng l y n o n n o r m a l .

T h i s i n f l a t i o n w a s e x a c e r b a t e d w i t h i n c r e a s i n g

s a m p l e s iz e . F o r e x a m p l e , n e a r l y h a l f o f t h e c o r -

r e c t l y s p e c i f ie d m o d e l s w e r e r e j e c t e d f o r N = 1 ,0 0 0

u n d e r t h e s e v e r e ly n o n n o r m a l c o n d i ti o n . U n d e r

m u l t i v a r i a t e n o r m a l i t y , t h e A D F X 2 w a s i n f l a t e d

a t s m a l l s a m p l e s i ze s , f o r e x a m p l e , r e j e c t i n g 4 3

o f t h e c o r r e c t l y s p e c if i e d m o d e l s a t N = 1 0 0. T h e

p e r f o r m a n c e o f t h e A D F i m p r o v e d w i t h in c r e a si n g

s a m p l e s i ze , b u t e v e n u n d e r m u l t i v a ri a t e n o r m a l i t y

a t N = 1 ,0 0 0, 1 0 o f t h e c o r r e c t m o d e l s w e r e

r e j e c t e d . T h e A D F X 2 w a s a l s o p o s i t i v e l y b i a s e d

w i t h i n c r e a s i n g n o n n o r m a l i t y , b u t t h i s b i a s w a s

a t t e n u a t e d w i t h i n c r e a s i n g s a m p l e s iz e . F i n a l ly , t h e

S B X 2 w a s v e r y w e l l b e h a v e d a t n e a r l y a l l s a m p l e

s i z e s a c r o s s a l l d i s t r i b u t i o n s . F o r e x a m p l e , a t a

s a m p l e s iz e o f N = 2 0 0 u n d e r s e v e r e n o n n o r m a l i t y ,

t h e S B X 2 r e j e c t e d 7 o f th e p r o p e r l y s p e c i f ie d

m o d e l s ( c o m p a r e d t o 2 5 f o r A D F a n d 3 6 f o r

M L ) . U n d e r t h e s e c o n d i t io n s , t h e p e r f o r m a n c e o f

t h e S B X 2 r e p r e s e n t e d a d i s t in c t i m p r o v e m e n t o v e r

t h e M L X 2 u n d e r c o n d i t io n s o f n o n n o r m a l i t y . I n -

t e r e st i n g ly , t h e S B a n d A D F p e r f o r m e d s i m i la r ly

a t s a m p l e s o f N = 5 0 0 a n d N = 1 , 00 0 .

M o d e l S p e c i f ic a t io n 2 .

T h e r e s u l ts f r o m M o d e l

2 a r e p r e s e n t e d i n T a b l e 2 . R e c a l l t h a t M o d e l 2

e s t i m a t e d t w o f a c t o r l o a d i n g s i n t h e s a m p l e t h a t

d i d n o t e x i s t i n t h e p o p u l a t i o n . B e c a u s e t h e e r r o r

i s t h e a d d i t i o n o f t w o t r u l y n o n e x i s t e n t p a r a m e -

t e r s , t h i s c a n b e c o n s i d e r e d a p r o p e r l y s p e c i f i e d

m o d e l , a n d t h e e x p e c t e d v a l u e o f th e m o d e l c h i -

s q u a r e w a s e q u a l t o t h e m o d e l d e g r e e s o f f r e e -

d o m f o r a l l e s t i m a t o r s a c r o s s a l l s a m p l e s i z e s ,

E X 2) = 22.0.

O v e r a l l , t h e r e s u l t s f r o m M o d e l 2 c l o s e ly fo l -

l o w e d t h o s e o f M o d e l 1 . U n d e r m u l t i v a r i a te n o r -

m a l i ty , t h e r e j e c ti o n r a t e s f o r b o t h t h e M L a n d

S B w e r e s l i g h tl y h i g h e r t h a n e x p e c t e d a t N = 1 0 0

b u t w e r e u n b i a s e d a t N = 2 0 0 a n d g r e a t e r . U n d e r

m u l t i v a r ia t e n o r m a l i t y , th e A D F a g a in r e j e c t e d a

v e r y h ig h n u m b e r o f m o d e l s a t t h e t w o s m a l l e r

s a m p l e s i z e s b u t w a s u n b i a s e d a t t h e t w o l a r g e r

s a m p l e s iz e s. A s w i t h M o d e l 1 , t h e M L X 2 w a s

3 Al l im p roper so lu tions (non converg ed so lu t ions and

so lu t ions tha t converged bu t re su l t ed in ou t -o f -bound

param ete rs , e . g . , Heywood cases ) were d ropped f rom

subsequent analyses . Collaps ing across a l l condit ions ,

90 of the rep l i ca t ions were p rope r fo r M L and 83

w e r e p r o p e r f o r A D F .


7/14

2 2 C U R R A N , W E S T , A N D F I N C H

T a b l e 1

Observed C hi-Square Expected Chi-Square Percentage Bias and Percentage of Rejected Models

for Model Specif ication 1

No rma l M o d e ra t e l y n o n n o rma l Se v ere ly n o n n o rm a l

Ob se rv e d Ex p e c t e d % % Ob se rv e d Ex p e c t e d % % Ob se rv e d Ex p e c t e d % %

Size X2 X2 X2 Bias Rejec t X2 X2 Bias Rejec t X2 X2 Bias Rejec t

100 M L 25.01 24.0 4.0 5.5 29.35 24.0 22.0 20.0 33.54 24.0 40.0 30.0

SB 25.87 24.0 8.0 7.5 26.06 24.0 9.0 8.5 27.26 24.0 14.0 13.0

A D F 36.44 24.0 52.0 43.0 38.04 24.0 59.0 49.0 44.82 24.0 87.0 68.0

200 ML 24.78 24.0 3.0 6.5 30.15 24.0 26.0 25.0 34.40 24.0 43.0 36.0

SB 25.22 24.0 5.0 8.5 25.44 24.0 6.0 8.0 25.80 24.0 8.0 6.5

A D F 29.19 24.0 22.0 19.0 29.27 24.0 22.0 19.0 31.29 24.0 30.0 25.0

500 ML 23.94 24.0 0.0 3.5 31.26 24.0 30.0 24.0 35.55 24.0 48.0 40.0

SB 24.10 24.0 0.0 5.0 25.44 24.0 6.0 6.9 24.85 24.0 4.0 8.5

A D F 25.92 24.0 8.0 11.0 26.42 24.0 10.0 6.7 26.83 24.0 12.0 8.5

1000 ML 25.05 24.0 4.0 7.0 30.78 24.0 28.0 24.0 37.40 24.0 56.0 48.0

SB 25.16 24.0 5.0 8.0 24.77 24.0 3.0 7.5 25.01 24.0 4.0 7.0

A D F 25.79 24.0 7.0 9.5 25.36 24.0 6.0 7.5 25.47 24.0 6.0 7.2

Note.

U n i v a r i a t e s k e w n e s s a n d k u r t o s e s w e r e ( 0 ,0 ) , (2 , 7) , a n d ( 3 ,2 1) f o r n o r m a l , m o d e r a t e l y n o n n o r m a l , a n d s e v e r e l y n o n n o r m a l

d i s t r ib u t i o n s , r e s p ec t iv e l y . M L = m a x i m u m l i k e l i ho o d ; S B = S a t o r r a - B e n t l e r r e s c a le d ; A D F = a s y m p t o t i c d i s t r ib u t i o n

f re e .

i n c r e a s i n g l y p o s i t i v e l y b i a s e d w i t h i n c r e a s i n g n o n -

n o r m a l i t y , a n d t h i s i n f l a t i o n w a s e x a c e r b a t e d w i t h

i n c r e a s i n g s a m p l e s iz e . I n c o m p a r i s o n , t h e S B X 2

s h o w e d m i n i m a l b i a s w i t h i n c r e a s i n g n o n n o r m a l -

i ty , a l t h o u g h t h e o b s e r v e d r e j e c t i o n r a t e s a t t h e

s m a l l e s t s a m p l e s i ze w e r e s l i gh t l y l a r g e r t h a n e x -

p e c t e d . E v e n u n d e r s e v e r e n o n n o r m a l i t y , t h e S B

X 2 a g a i n s h o w e d l i t t l e e v i d e n c e o f b i a s , e s p e c i a l l y

a t s a m p l e s i z e s o f N - - 2 0 0 o r g r e a t e r . F i n a l l y ,

t h e A D F X 2 w a s p o s i t i v e l y b i a s e d w i t h i n c r e a s i n g

n o n n o r m a l i t y a t t h e s m a l l e r s a m p l e s i z es b u t w a s

u n b i a s e d a t s a m p l e s i z e s o f N = 5 0 0 a n d N =

1 ,0 0 0 , e v e n u n d e r s e v e r e n o n n o r m a l i t y .

M o d e l S p e c i f i c a t i o n 3 .

M o d e l S p e c i f i c a t i o n 3

e x c l u d e d t w o f a c t o r l o a d i n g s i n t h e s a m p l e ( )t =

. 35 ) t h a t t r u l y e x i s t e d i n th e p o p u l a t i o n . T h e s e

r e s u l t s a r e p r e s e n t e d i n T a b l e 3 . R e c a l l t h a t d u e

t o t h e e x c l u s i o n o f e x i s t i n g p a r a m e t e r s , t h e r e w a s

a d i f f e r e n t e x p e c t e d v a l u e f o r e a c h t e s t s t a ti s t i c .

A l s o , b e c a u s e t h e m o d e l w a s m i s s p e c i f i e d i n t h e

T a b l e 2

Observed C hi-Square Expected C hi-Square Percentage Bias and Percentage of Rejected Models


No rma l M o d e ra t e l y n o n n o rm a l Se ve rel y n o n n o rma l

Ob se rv e d Ex p e c t e d % % Ob se rv e d Ex p e c t e d % % Ob se rv e d Ex p e c t e d % %

Size X2 g 2 X2 Bias Rejec t X2 2 2 Bias Rejec t X2 X2 Bias Rejec t

100 M L 23.42 22.0 6.0 9.6 26.89 22.0 22.0 22.2 29.82 22.0 36.0 34.4

SB 24.19 22.0 9.0 12.1 23.80 22.0 8.0 8.5 25.07 22.0 14.0 11.1

A D F 31.0 22.0 41.0 30.5 34.95 22.0 59.0 40.8 46.45 22.0 111.0 63.2

200 ML 22.48 22.0 2.0 6.0 27.70 22.0 26.0 26.5 31.77 22.0 44.0 36.7

SB 22.86 22.0 3.0 7.0 23.75 22.0 8.0 8.5 24.10 22.0 10.0 8.5

A D F 26.43 22.0 20.0 17.5 26.06 22.0 18.0 14.5 28.52 22.0 30.0 22.0

500 M L 21.89 22.0 0.0 6.0 26.68 22.0 21.0 19.0 31.86 22.0 45.0 32.5

SB 22.02 22.0 0.0 7.0 21.90 22.0 0.0 5.5 23.33 22.0 6.0 6.5

A D F 23.13 22.0 5.0 7.0 23.04 22.0 5.0 8.0 24.02 22.0 9.0 5.5

1000 M L 22.25 22.0 0.0 5.0 26.05 22.0 18.0 14.5 33.37 22.0 52.0 41.5

SB 22.31 22.0 0.0 3.5 21.14 22.0 4~0 4.0 22.74 22.0 3.0 7.0

A D F 22.09 22.0 0.0 6.0 23.32 22.0 6.0 9.5 23.41 22.0 6.0 7.5

Note.

U n i v a r i a t e s k e w n e s s a n d k u r t o s e s w e r e ( 0 ,0 ) , ( 2, 7 ), a n d ( 3 ,2 1) f o r n o r m a l , m o d e r a t e l y n o n n o r m a l , a n d s e v e r e l y n o n n o r m a l

d i s t ri b u t io n s , r e s p e c ti v e ly . M L = m a x i m u m l i k e l i h o o d; S B = S a t o r r a B e n t l e r r es c a l ed ; A D F = a s y m p t o t i c d i s t r ib u t i o n

free .


8/14


T a b l e 3

Observed C hi-Square Expected C hi-Square Percentage Bias and Percentage of Rejected Models

for Model Specification 3

Normal Moderately nonnormal Severely nonnormal

Observed Expected Observed Expected Observed Expected

Size X2 2,2 X2 Bia s Reject X2 X2 Bias Reject A, X2 Bias Reject

100 M L 38.45 37.62 2.0 54.3 46.50 37.75 23.0 78.9 52.87 37.77 40.0 81.4

SB 39.63 37.65 5.0 57.9 37.63 33.99 11.0 49.1 38.10 30.85 24.0 47.1

AD F 52.04 33.74 54.0 80.3 60.88 29 .68 105 .0 86.3 81.31 27 .29 19 8.0 95.3

200 M L 51.07 51.38 0.0 88.1 59.72 51.66 16.0 93.7 68.58 51.68 33.0 95.4

SB 51.65 51.44 0.0 89.9 47.04 44.09 7.0 82.4 44.66 37.77 18.0 78.9

A D F 50.17 43.59 15.0 85.6 47.93 35.42 36.0 84.8 47.16 30.61 54.0 75.2

500 ML 92.27 92.66 0.0 100.0 99.39 93.37 6.0 1 0 0 . 0 109.87 93.42 18 .0 100.0

SB 92.52 92.80 0.0 100 .0 75.04 74.40 1.0 100 .0 63.46 58.52 8.0 97.2

A D F 76.89 73.11 5.0 100 .0 60.10 52.63 14.0 99.5 53.67 40.58 32.0 96.7

1000 ML 161.46 161.35 0.0 1 0 0 .0 171.07 162.88 5.0 10 0. 0 180.90 162.98 11.0 100.0

SB 161.66 161.74 0.0 1 0 0 .0 126 .23 124.89 2.0 1 0 0 .0 101.25 93.11 9.0 100.0

A D F 127.71 122.32 4.0 10 0.0 90.23 81.31 11 .0 1130.0 76.10 57.20 33 .0 100.0

Note. Univ ariate skewness and k urtoses were (0,0) , (2,7), and (3,21) for normal, mo derately nonnorma l, and severely nonnorm al

distr ibutions, respectively. ML = maximum likel ihood; SB = Satorra -Ben tler rescaled; A DF = asymp totic distr ibution free.

s a m p l e , t h e p e r c e n t a g e o f r e j e c t e d m o d e l s w a s n o

l o n g e r a m e a n i n g f u l g u i d e w i t h w h i c h t o ju d g e t h e

b e h a v i o r o f t h e t e s t s t a ti s t ic s . T h u s , t h e f o l l o w i n g

r e s u l ts w i ll n o w b e p r e s e n t e d i n t e r m s o f t h e r e l a -

t i v e b i a s i n t h e t e s t s t a t i s t i c s . 4

U n d e r m u l t i v a r i a t e n o r m a l i t y , t h e e x p e c t e d v a l -

u e s f o r t h e M L a n d S B X 2 w e r e n e a r l y i d e n t i c a l

a c r o s s a l l f o u r s a m p l e s i ze s . T h i s i s f u r t h e r s u p p o r t

t h a t f o r n o r m a l d i s t r ib u t i o n s , n o s c a l i n g c o r re c t i o n

i s r e q u i r e d f o r t h e M L X 2, a n d t h e S B X 2 t h u s

s i m p l i f i e s t o t h e M L X 2. A d d i t i o n a l l y , n e i t h e r t h e

M L o r S B t e s t s t a t i s t ic s h o w e d a p p r e c i a b l e b i a s

u n d e r n o r m a l i t y a c r o s s a l l f o u r s a m p l e s i z e s. I n

c o m p a r i s o n , t h e e m p i r i c a l e s t i m a t e o f t h e e x -

p e c t e d v a l u e o f t h e A D F X 2 w a s s m a l l e r t h a n t h a t

o f t h e M L o r S B t e s t s t a ti s t ic s . R e c a l l t h a t t h e

e x p e c t e d v a l u e f o r a l l t h r e e t e s t s ta t i s ti c s w e r e

e q u a l f o r t h e p r o p e r l y s p e c i fi e d m o d e l s . T h e l o w e r

e x p e c t e d v a l u e o f th e A D F f o r m i s sp e c i f ie d m o d -

e ls e v e n u n d e r m u l t i v a r i a t e n o r m a l i t y s u g g e s t s th a t

t h i s t e s t s t a t is t i c m a y h a v e l e s s p o w e r t o r e j e c t t h e

n u l l h y p o t h e s i s c o m p a r e d w i t h t h e M L o r S B X 2.

U n l i k e t h e M L a n d S B , t h e A D F w a s s ig n i f i c a n tl y

p o s i t i v e ly b i a s e d a t t h e t w o s m a l l e r s a m p l e s iz e s.

F o r e x a m p l e , a t N = 1 00 t h e a v e r a g e o b s e r v e d

A D F X 2 w a s 5 4 l a r g e r t h a n t h e e x p e c t e d v a l u e .

T h i s b i a s d r o p p e d t o 1 5 a t N = 2 0 0 a n d w a s

n e g l i g i b l e a t th e t w o l a r g e r s a m p l e s i z e s.

T h e f in d in g s b e c o m e m o r e c o m p l i c a t e d g i ve n

n o n n o r m a l d i s t r i b u t io n s . T h e e x p e c t e d v a l u e o f

t h e M L X 2 w a s t h e s a m e a c r o s s a l l t h r e e d i s t r i b u -

t io n s . A s w i t h t h e p r e v i o u s m o d e l s , t h e M L X 2

s h o w e d i n c r e a s i n g l e v e l s o f p o s i t i v e b i a s w i t h i n -

c r e a s i n g n o n n o r m a l i t y . A p a r t i c u l a r l y i n t e r e s t i n g

f i n d in g p e r t a i n e d t o t h e e x p e c t e d v a l u e s o f th e S B

a n d A D F t e s t s t at i st i cs u n d e r n o n n o r m a l i t y . B o t h

t h e e x p e c t e d a n d t h e o b s e r v e d v a l u e s o f t h e S B

a n d A D F t e s t s t a ti s t i c s d e c r e a s e d w i t h i n c r e a s i n g

n o n n o r m a l i t y . F o r e x a m p l e , a t s a m p l e s iz e N =

2 0 0 , th e e x p e c t e d v a l u e f o r t h e S B X 2 w a s a p p r o x i -

m a t e l y 5 1 u n d e r n o r m a l i t y , 4 4 u n d e r m o d e r a t e

n o n n o r m a l i t y , a n d 3 8 u n d e r s e v e r e n o n n o r m a l i t y .

T h e A D F t e s t s t a t is t i c s h o w e d a s i m i l a r p a t t e r n .

T h e d i r e c t i n t e r p r e t a t i o n o f t h i s f i n d i n g is t h a t i t

i s i n c r e a s i n g l y d i f f i c u lt to d e t e c t a m i s s p e c i f i c a t i o n

w i t h i n t h e m o d e l g i v e n t h e a d d e d v a r i a b i l i t y d u e

t o t h e n o n n o r m a l d i s t r i b u t io n o f t h e d a ta . T h u s ,

t h e p o w e r o f S B a n d A D F t e s t s t at i st ic s d e c r e a s e d

w i t h i n c r e a s in g n o n n o r m a l i t y .

U n d e r m o d e r a t e n o n n o r m a l i t y , t h e S B X 2 w a s

s l i g h t l y b i a s e d a t N = 1 00 ( 1 1 ) b u t w a s u n b i a s e d

a t s a m p l e s i z e s o f N = 2 0 0 a n d a b o v e . I n c o m p a r i -

s o n , a l so u n d e r m o d e r a t e n o n n o r m a l i t y , t he A D F

X 2 s h o w e d e x t r e m e b i a s a t t h e s m a l l e r s a m p l e s iz e s

4 M o d e l s 1 a n d 2 c o u l d h a v e s i m i l a r l y b e e n e v a l u a t e d

u s i n g th e p e r c e n t a g e o f b ia s , a n d t h e s a m e c o n c l u s i o n s

w o u l d h a v e b e e n d r a w n . T h e p e r c e n t a g e o f r e j e c te d

m o d e l s w a s c h o s e n i n s t e a d f o r M o d e l s 1 a n d 2 g i v e n

t h e m o r e d i r e c t i n t e r p r e t a b i l i t y o f th e f i n di n g s .


9/14


T a b l e 4

Observed Ch i-Square Expected C hi-Square Percentage Bias and Percentage of Rejected Models


Normal Moderately nonnormal Severely nonnormal

Observed Expected % % Observed Expected % % Observed Expected % %

Size X2 X2 X Bias Reject X ~v Bias Reject X2 X Bia s Reject

100 ML 34.03 32.52 5.0 48.7 40.37 32.52 24.0 63.4 45.15 32.45 39.0 78.6

SB 34.97 32.56 7.0 51.8 33.94 29.76 14.0 45.5 34.09 27.33 25.0 50.3

AD F 47.99 30.66 56.0 82.1 48.67 27.19 79.0 85.4 63.25 25 .06 152.0 91.8

200 ML 43.75 43.14 1.0 81.1 49.84 43.14 16.0 91.0 58.14 43.01 35.0 88.1

SB 44.34 43.23 3.0 81.6 39.55 37.59 5.0 67.9 38.48 32.71 18.0 60.4

A D F 46.42 39.41 18.0 85.5 41.84 32.43 29.0 73.4 46.05 28.16 64.0 84.5

500 ML 75.29 75.04 0.0 100.0 81.35 75.04 8.0 100.0 91.71 74.68 23. 0 100.0

SB 75.62 75.23 0.0 100 .0 62.09 61.09 2.0 97.0 55.94 48.85 15.0 92.6

A D F 69.62 65.66 6.0 100 .0 53.84 48.15 12.0 94.2 49.13 37.45 31.0 93.2

1000 ML 128.71 128.20 0.0 1 0 0 .0 133.8 6 128.20 4.0 1 0 0 .0 144 .56 127.46 13.0 100.0

SB 129.16 128.56 0.0 1 0 0 .0 10 0.4 8 100.26 0.0 100.0 83.44 75.76 1 0.0 100.0

AD F 111.39 109.41 2.0 100.0 80.91 74.35 9.0 100.0 68.25 52.92 30 .0 100.0

Note.

Univ ariate skewness and kurtoses were (0,0) , (2,7), and (3,21) for normal, mode rately nonnormal, and severely nonnormal

distr ibutions, respectively. ML = maximum likel ihood; SB = Satorra-B entler rescaled; AD F = asymptotic distr ibution free.

a n d r e m a i n e d b i a s e d e v e n a t N = 1 ,0 0 0 ( 1 1 % ) .

U n d e r s e v e r e n o n n o r m a l i t y , t h e S B X 2 s h o w e d

s u b s t a n t i a l b i a s a t t h e t w o s m a l l e r s a m p l e s i z e s

( e .g . , 1 8 % a t N = 2 0 0 ) b u t w a s o n l y m o d e r a t e l y

b i a s e d a t t h e l a r g e r s a m p l e s i z e s ( e .g . , 9 % a t N =

1 ,0 0 0 ). T h e A D F s h o w e d v e r y h i g h l e v e l s o f r e l a -

t i v e b i a s a c r o s s a l l f o u r s a m p l e s i z e s u n d e r s e v e r e

n o n n o r m a l i t y a n d w a s o v e r e s t i m a t e d b y 3 3 % e v e n

a t t h e l a r g e s t s a m p l e s i z e N = 1 , 00 0 .

Model Spec i f i ca t i on 4 .

M o d e l S p e c i f ic a t i o n 4

c o n t a i n e d b o t h e r r o r s o f i n cl u s i o n a n d e x c l u s io n .

T w o c r o s s - lo a d i n g s e x i st e d in t h e p o p u l a t i o n t h a t

w e r e n o t e s t i m a t e d i n th e s a m p l e ( A = . 35 ), a n d

t w o c r o s s - l o a d i n g s w e r e e s t i m a t e d i n t h e s a m p l e

t h a t d i d n o t e x i s t i n t h e p o p u l a t i o n ( A = 0 ) . T h e s e

r e s u l t s a r e p r e s e n t e d i n T a b l e 4 .

T h e f i n d i n g s f r o m M o d e l S p e c i f i c a t i o n 4 fo l -

l o w e d t h e s a m e g e n e r a l p a t t e r n a s w a s o b s e r v e d

f o r M o d e l S p e c i f ic a t i o n 3 . T h e p r i m a r y d i f f e r e n c e

w a s t h a t t h e e x p e c t e d v a l u e s a n d r e j e c t i o n r a te s

i n M o d e l 4 w e r e l o w e r c o m p a r e d w i t h t h o s e o f

M o d e l 3 . T h i s r e s u l t m a y i n i ti a l ly a p p e a r c o u n t e r -

i n t u it i v e gi v e n th a t M o d e l 4 c o m b i n e d e r r o r s o f

b o t h i n c l u si o n a n d e x c l u s io n . H o w e v e r , u n l i k e

M o d e l 3 , M o d e l 4 c o n t a i n e d t h e s i m u l t a n e o u s e s t i-

m a t i o n o f t h e t w o t r u ly n o n e x i s t e n t p a t h s a n d t h e

e x c l u s io n o f t h e t w o t r u ly e x i s t e n t p a th s . T h e m e a n

e s t i m a t e d f a c t o r l o a d i n g s f o r t he t w o a d d i t i o n a l

p a t h s i n M o d e l 2 ( w h e r e n o o t h e r p a t h s w e r e e x -

c l u d e d ) w a s A = 0 ( th e p o p u l a t i o n e x p e c t e d v a l u e ) .

H o w e v e r , t h e m e a n f a c t o r l o a d i n g s f o r t h e s e s a m e

t w o a d d i t i o n a l p a th s i n M o d e l 4 w a s h = - . 4 0 .

T h u s , t h e s e a d d i t i o n a l fr e e p a r a m e t e r s s e r v e d t o

a b s o r b t h e m i s sp e c i f i c a t io n , a n d M o d e l 4 r e -

s u i t e d i n a b e t t e r f it t o t h e d a t a t h a n d i d M o d e l 3 .

A s w i t h M o d e l 3 , u n d e r m u l t i v a r i a t e n o r m a l i t y ,

t h e e x p e c t e d v a l u e s o f t h e M L a n d S B X 2 w e r e

e q u a l t o o n e a n o t h e r w h e r e a s t h e e x p e c t e d v a l u e

o f th e A D F g 2 w a s s m a l l e r . N e i t h e r t h e M L o r S B

X 2 s h o w e d a n y s i g n i f i c a n t b ia s u n d e r t h e n o r m a l

d i s t r i b u t i o n a c r o s s a n y o f t h e f o u r s a m p l e s i z e s.

T h e l a r g e s t b i a s w a s f o r t h e S B X 2 a t N = 1 0 0

( 7 % ), b u t t h e m a g n i t u d e o f b i a s d r o p p e d t o n e a r

0 a t s a m p l e s iz e s o f N = 2 0 0 a n d a b o v e . I n c o m p a r i -

s o n , t h e A D F X 2 w a s a g a i n s i g n i f ic a n t l y o v e r e s t i -

m a t e d a t th e t w o s m a l l e r s a m p l e s i z es b u t w a s

u n b i a s e d a t s a m p l e s i ze s o f N = 5 0 0 a n d a b o v e .

T h e e x p e c t e d v a l u e o f t h e M L X 2 w a s a g a i n

e q u a l a c r o s s d i s t r i b u t i o n s , a n d t h e M L X 2 w a s i n -

c r e a s i n g ly p o s i t i v e l y b i a s e d w i t h i n c r e a s i n g n o n -

n o r m a l i t y . L i k e M o d e l 3 , t h e e x p e c t e d v a l u e s f o r

t h e S B a n d A D F X 2 d e c r e a s e d w i t h i n c r e a s i n g

n o n n o r m a l i t y . T h e S B g 2 s h o w e d i n c r e a s i n g p o s i-

t i v e b i a s w i t h i n c r e a s i n g n o n n o r m a l i t y . T h i s b i a s

b e c a m e n e g l i g i b le a t N = 2 00 u n d e r m o d e r a t e

n o n n o r m a l i t y ( 5 % ) b u t w a s s t i ll s l i g h t ly b i a s e d

e v e n a t N = 1 ,0 0 0 u n d e r s e v e r e n o n n o r m a l i t y

( 1 0 % ) . F i n a l l y , t h e A D F X 2 w a s a g a i n s t r o n g l y

b i a s e d , w i t h i n c r e a s i n g n o n n o r m a l i t y e v e n a t th e

l a r g e s t s a m p l e s i z e .


10/14

C H I - S Q U A R E T E S T S T A T IS T I C S 2 5

D i s c u s s i o n

T h e f ir st t w o m o d e l s w e r e t h e o r e t i c a l l y p r o p -

e r l y s p ec i f ie d . M o d e l 1 w a s e s t i m a t e d i n t h e

s a m p l e p r e c i s e l y a s i t e x i s t e d i n t h e p o p u l a t i o n ,

w h e r e a s M o d e l 2 i n c l u d e d t w o p a r a m e t e r s i n t h e

s a m p l e t h a t d i d n o t e x i s t i n t h e p o p u l a t i o n . T h e

f i n d i n g s f o r t h e M L X 2 f r o m M o d e l s 1 a n d 2

c l o s e l y r e p l i c a t e b o t h p r e v i o u s t h e o r e t i c a l p r e d i c -

t i o n s a n d e m p i r i c a l f i n d i n g s . F o r e x a m p l e , t h e

M L X 2 s h o w e d n o e v i d e n c e o f b i a s a c r o s s a l l

s a m p l e s i z e s u n d e r m u l t i v a r i a t e n o r m a l d i s t r i b u -

t i o n s b u t w a s s i g n i f ic a n t ly i n f l a t e d w i t h i n c r e a s i n g

n o n n o r m a l i t y . T h u s , a c o r r e c t m o d e l w a s s i g n if i-

c a n t l y m o r e l i k e l y t o b e e r r o n e o u s l y r e j e c t e d

b a s e d o n t h e M L X 2 g i v e n d e p a r t u r e s f r o m a

m u l t i v a r i a t e n o r m a l d i s t r i b u t i o n t h u s r e s u l t i n g

i n a n i n c r e a s e d T y p e I e r r o r r a t e ) .

T h e A D F a n d S B t e s t s ta ti st ic s h a v e b e e n p r o -

P O se d as a l t e r n at i v e s t o t h e n o r m a l t h e o r y M L t e st

s ta t is ti c w h e n t h e o b s e r v e d d a t a d o n o t m e e t t h e

m u l t i v a r i a t e n o r m a l i t y a s s u m p t i o n . C o n s i s t e n t

w i t h p r e v i o u s r e s e a r c h o n p r o p e r l y s p e c i fi e d m o d -

e l s, t h e A D F X 2 w a s s u b s t a n t i a ll y i n f l a t e d a t

s m a l l e r s a m p l e s i z es , e v e n u n d e r m u l t i v a r i a t e n o r -

m a l d i s t r i b u t io n s . A l t h o u g h t h is s m a l l s a m p l e s i ze

i n f l a t i o n w a s e x a c e r b a t e d w i t h i n c r e a s i n g n o n n o r -

m a l i t y , t h e A D F w a s u n b i a s e d a t s a m p l e s i z e s o f

N = 5 0 0 a n d a b o v e , r e g a r d l e s s o f d i st r i b u t i o n .

T h e S B X 2 p e r f o r m e d q u i t e w e l l a c r o ss n e a r l y a l l

s a m p l e s i z e s a n d a l l d i s t r i b u t i o n s a n d s h o w e d n o

e v i d e n c e o f bi as e v e n u n d e r s e v e r e l y n o n n o r m a l

d i s t r i b u t i o n s a t s a m p l e s i z e s o f N = 2 0 0 o r m o r e .

T h e s e a r e v e r y h e a r t e n i n g f i n d in g s f o r t h e p r a c t i c-

i n g r e s e a r c h e r w h o e n c o u n t e r s n o n n o r m a l d a t a

a s a w a y o f li f e e . g. , i n t h e s t u d y o f a d o l e s c e n t

s u b s t a n c e u s e o r p s y c h o p a t h o l o g y ) . N o t o n l y w a s

t h e S B X 2 a c c u r a t e u n d e r e v e n s e v e r e l y n o n n o r m a l

d i s t r i b u ti o n s , b u t t h e S B X 2 s i m p l i fi e d t o t h e M L

X 2 u n d e r c o n d i t i o n s o f m u l t i v a ri a t e n o r m a l i ty . A s -

s u m i n g a p r o p e r l y s p e c i f ie d m o d e l , t h e S B X 2 a p -

p e a r s t o b e a v e r y u s e f ul m e a s u r e o f f it g i v e n m o d -

e r a t e l y si z ed s a m p l e s a n d n o n n o r m a l d a t a.

W h e r e a s m a n y o f t h e re s u lt s f r o m M o d e l s 1

a n d 2 w e r e p r e d i c t e d f r o m t h e o r y a n d p r e v i o u s

r e s e a r c h , t h e f i n d i n g s f r o m M o d e l s 3 a n d 4 w e r e

n o t . R e c a l l t h a t M o d e l s 3 a n d 4 w e r e t w o v a r i a t i o n s

o f a m i ss p e c if i ed m o d e l w h e r e t h e m o d e l e s t i m a t e d

i n t h e s a m p l e d i d n o t c o n f o r m t o t h e m o d e l t h a t

e x i s t e d i n t h e p o p u l a t i o n . S t u d y i n g t h e b e h a v i o r

o f t h e t e s t s t a t is t ic s u n d e r t h e s e c o n d i t i o n s i s o f

p a r t i c u l a r i n t e r e s t g i v e n t h e h i g h l i k e l i h o o d t h a t

t h e m o d e l e s t i m a t e d i n t h e s a m p l e d o e s n o t p r e -

c i s e ly c o n f o r m t o t h e m o d e l t h a t e x i st s i n t h e p o p u -

l a t io n . T h e r e s u l t s f o r t h e M L X 2 w e r e a s e x p e c t e d :

T h e M L t e s t st a ti s ti c s h o w e d n o e v i d e n c e o f b i as

a t a n y s a m p l e s i z e u n d e r m u l t i v a r i a t e n o r m a l i t y

b u t w a s i n c r e a s i n g l y in f l a t e d g i v e n i n c r e a si n g n o n -

n o r m a l i t y . A s i n M o d e l s 1 a n d 2 , t h e S B X 2 a l so

s h o w e d n o e v i d e n c e o f b ia s a t a n y s a m p l e s iz e

g i v e n m u l t i v a r i a t e n o r m a l i t y , a n d t h u s s i m p l i f i e d

t o t h e M L X :. I n t e r e s ti n g l y , t h e e x p e c t e d v a l u e

f o r t h e A D F X 2 u n d e r m o d e l m i s s p ec i fi c a ti o n w a s

m u c h s m a l l er t h a n t h a t o f t h e M L a n d S B , e v e n

u n d e r m u l t i v a r i a t e n o r m a l i t y . T h i s s u g g e s t s t h a t ,

c o m p a r e d t o t h e M L a n d S B , t h e A D F t e st st a ti s ti c

m a y b e a l es s p o w e r f u l t e s t o f t h e n u l l h y p o t h e s i s .

T h i s c o n c l u s i o n i s t e n t a t i v e , a n d m o r e w o r k i s

n e e d e d t o b e t t e r u n d e r s t a n d t h i s f i n d i n g . L i k e

M o d e l s 1 a n d 2 , t h e A D F w a s p o s i ti v e ly b ia s e d

u n d e r m u l t i v a r i a t e n o r m a l d i s t r i b u t i o n s a t t h e t w o

s m a l l e r s a m p l e s i z e s b u t s h o w e d n o b i a s a t t h e t w o

l a r g e r s a m p l e s i z e s .

T h e m o s t s u r p r i s i n g f i n d i n g s r e l a t e d t o t h e b e -

h a v i o r o f th e S B a n d A D F t e s t s t a ti s ti c s u n d e r t h e

s i m u l t a n e o u s c o n d i t i o n s o f m i s s p e c i fi c a t i o n a n d

m u l t i v a r ia t e n o n n o r m a l i t y M o d e l s 3 a n d 4 ). T h e

e x p e c t e d v a l u e s o f t h e s e t e s t s t a t i s t i c s m a r k e d l y

d e c r e a s e d w i t h i n c r e a s i n g n o n n o r m a l i t y . T h a t i s ,

a ll e ls e b e i n g e q u a l , t h e S B a n d A D F t e s t s ta t is t ic s

w e r e l es s l ik e l y to d e t e c t a s p e c i f ic a t i o n e r r o r g i v e n

i n c re a s in g d e p a r t u r e s f r o m a m u l t i v a r ia t e n o r m a l

d i s t r i b u t i o n . T h e m o r e s e v e r e t h e n o n n o r m a l i t y ,

t h e g r e a t e r t h e c o r r e s p o n d i n g l o ss o f p o w e r . T h i s

r e s u l t w a s u n e x p e c t e d , a n d w e a r e n o t a w a r e o f

a n y p r e v i o u s d i s c u s s i o n s o f t h i s f i n d i n g .

A l t h o u g h t h e s p e c i f ic r e a s o n f o r t h i s l os s o f

p o w e r i s c u r r e n t l y n o t k n o w n , w e t h e o r i z e t h a t i t

i s d u e t o t h e i n c lu s i on o f t h e f o u r t h - o r d e r m o m e n t s

k u r t o s e s ) in t h e c o m p u t a t i o n o f th e S B a n d A D F

t e s t s t a t i s t i c s , i n f o r m a t i o n t h a t i s i g n o r e d b y t h e

n o r m a l t h e o r y M L X : . R e c a l l t h a t a n o r m a l d i s t r i -

b u t i o n i s c o m p l e t e l y d e s c r i b e d b y t h e f i r st t w o

m o m e n t s , t h e m e a n a n d t h e v a r i a n ce . A s t h e d i s tr i-

b u t i o n b e c o m e s i n c r e a s i n g l y n o n s y m m e t r i c , i s

c h a r a c t e r i z e d b y t h ic k e r o r t h i n n e r t ai ls c o m p a r e d

w i t h t h e n o r m a l c u r v e ) , o r b o t h , a d d i ti o n a l p a r a m -

e t e r s a r e n e e d e d t o d e s c r i b e t h i s m o r e c o m p l e x

d i s t r i b u t io n . B e c a u s e M L i s a n o r m a l t h e o r y e s ti -

m a t o r , i t is a ss u m e d t h a t t h e f o u r t h - o r d e r m o -

m e n t s a r e e q u a l t o 0 , m u l t i v a r i a t e k u r t o s i s i s i g -

n o r e d , a n d t h e e x p e c t e d v a l u e o f t h e M L X 2 i s


11/14


e q u a l a c r o s s a l l d i s t r i b u t i o n s . 5 I n c o n t r a s t , t h e

A D F a n d S B d o n o t as s u m e m u l t i v a ri a t e n o r m a l -

i ty , t h e f o u r t h - o r d e r m o m e n t s a r e n o t a s s u m e d t o

e q u a l 0 , a n d m e a s u r e s o f m u l t i v a r i a t e k u r t o s i s a r e

e x p li c it ly in c o r p o r a t e d i n t o t h e c o m p u t a t i o n o f t h e

t e s t s t a t i s t i c s . A s a r e s u l t , t h e e x p e c t e d v a l u e s o f

t h e A D F a n d S B d i r e c tl y d e p e n d u p o n t h e p a r t ic u -

l a r c h a r a c t e r is t i c s o f t h e m u l t i v a r i a t e d i s t r i b u t i o n

u n d e r c o n s i d e r a ti o n .

T h e i n c l u s i o n o f m u l t i v a r i a t e k u r t o s i s i n t o t h e

c o m p u t a t i o n o f t h e S B a n d A D F t e s t s t a ti s ti c s

p r o v i d e s t h e c r i ti c a l i n f o r m a t i o n n e c e s s a r y t o f u l l y

d e s c r ib e t h e m o r e c o m p l e x n o n n o r m a l d i s t ri b u -

t i o n . H o w e v e r , t h i s a d d e d i n f o r m a t i o n r e s u l t i n g

f r o m t h e m o r e c o m p l e x d i s t r ib u t i o n a l so r e d u c e s

t h e a b i l it y o f t h e S B a n d A D F t o i d e n t i f y a g i v e n

m o d e l m i s s p e c i f i c a t i o n . O t h e r w i s e s t a t e d , w e c a n

t h i n k o f a h y p o t h e t i c a l s i gn a l t o n o i s e r a t i o i n

w h i c h t h e t e s t s t at i st i c is a t t e m p t i n g t o i d e n t i f y t h e

p r e s e n c e o f t h e s i g n a l (i . e. , t h e m i s s p e c i f i c a t i o n )

a g a i n s t t h e b a c k g r o u n d n o i s e ( i . e . , t h e s a m p l i n g

v a r ia b i li t y o f t h e d a ta ) . C o m p a r e d w i t h t h e n o r m a l

d i s t r i b u t io n , t h e n o n n o r m a l d i s t r i b u t i o n i s c h a r a c -

t e r i z e d b y a d d i t i o n a l n o i s e ( i n t h e f o r m o f n o n -

z e r o k u r t o s is ) t h a t m a k e s i t c o r r e s p o n d i n g l y m o r e

d i f fi c u lt t o i d e n t i f y t h e p r e s e n c e o f t h e s ig n a l. T h u s ,

a n y p a r t i c u l a r s i g n al i s e a s i e r t o d e t e c t g i v e n m u l t i -

v a r i a t e n o r m a l i t y t h a n i s t h e v e r y s a m e s i g na l g i v e n

t h e m u l t i v a r i a t e n o n n o r m a l d i s t r i b u t i o n s c o n s i d -

e r e d h e r e . T h e p o w e r o f t h e A D F a n d S B t e s t

s t a t i s t i c s ( a n d a n y t e s t s t a t i s t i c t h a t i n c o r p o r a t e s

i n f o r m a t i o n f r o m f o u r t h - o r d e r m o m e n t s ) t o d e t e c t

a g i v e n m i s s p e c i f i c a t i o n is t h u s d e c r e a s e d a s m u l t i -

v a r i a t e n o n n o r m a l i t y i n c r e a s es . 6 T h i s i n t e r p r e t a -

t i o n is o n l y sp e c u l a t i v e , a n d w e a r e c u r r e n t l y w o r k -

i n g o n d i s c e r n i n g p r e c i s e l y w h y t h i s lo s s o f p o w e r

u n d e r n o n n o r m a l i t y e x i s t s .

T h e r e a r e t w o i m p o r t a n t i m p l i c at i o ns o f t h e s e

f i n d in g s f o r t h e p r a c t i c i n g r e s e a r c h e r . F i r s t , t h e S B

X

w i ll a l m o s t a l w a y s b e s m a l l e r t h a n t h e M L X 2

u n d e r c o n d i t io n s o f m u l t i v a r i a te n o n n o r m a l i t y .

H o w e v e r , t h e l o w e r S B

X

d o e s n o t n e c e s s a r i l y

i m p l y t h a t t h e m o d e l i s a b e t t e r f i t t o t h e d a t a

b e c a u s e u n d e r n o n n o r m a l i t y t h e r e i s a s i m u l ta n e -

o u s d e c r e a s e i n t h e a b i l it y o f t h e S B X 2 t o d e t e c t

a m o d e l m i s s p e c i f ic a t i o n . T h e S B X 2 s s m a l l e r t h a n

t h e M L X 2 b e c a u s e o f t w o ( i n s e p a r a b l e ) r e a s o n s :

a c o r r e c t i o n f o r t h e i n f la t i o n t o t h e n o r m a l t h e o r y

M L X 2 a n d a d e c r e a s e i n s t a t i st i ca l p o w e r t o d e t e c t

a m i s s p e c i f i c a t i o n . T h e M L

X

a n d S B X 2 s h o u l d

t h u s b e i n t e r p r e t e d w i t h t h i s i n m i n d .

A s e c o n d i m p l i c a t i o n o f t h e s e f i n d i n g s i s t h a t i f

a r e s e a r c h e r i s p l a n n i n g a s t u d y t h a t w i l l n o t b e

c h a r a c t e r i z e d b y a m u l t i v a r i a t e n o r m a l d i s t r i b u -

t i o n , f u r t h e r s t e p s m u s t b e t a k e n t o c o m p e n s a t e

f o r t h e d e c r e a s e d s t a t i s t i c a l p o w e r t h a t r e s u l t s a s

a f u n c t i o n o f t h e n o n n o r m a l d a t a ( i .e ., p l a n t o

i n c l u d e a d d i t i o n a l s u b j e c t s i n t h e s t u d y ) . F o r e x -

a m p l e , t h e p o w e r e s t i m a t i o n m e t h o d s d e v e l o p e d

b y S a t o r r a a n d S a r i s ( 1 9 8 5 ) o n l y a p p l y t o n o r m a l

t h e o r y e s t i m a to r s . U s i n g th i s m e t h o d t o c o m p u t e

t h e r e q u i r e d s a m p l e s iz e n e e d e d t o a c h i e v e a g i v e n

l e v e l o f s t a ti s ti c a l p o w e r w i l l b e u n d e r e s t i m a t e d i f

t h e h y p o t h e s i z e d m o d e l w a s m i s s p e c i f i e d a n d

t e s t e d b a s e d o n d a t a t h a t d o n o t f o l l o w a m u l t i v a r i -

a t e n o r m a l d i s t r i b u t i o n .

R e c o m m e n d a t i o n s

O n t h e b a s is o f th e p r e v i o u s r e s u l ts , w e h a v e

s e v e r a l r e c o m m e n d a t i o n s f o r t h e p r a c t i c i n g r e -

s e a r c h e r . F i r s t, w e h a v e n o t i d e n t i f i e d a t w h a t p o i n t

t h e d a t a a p p r e c i a b l y d e v i a t e f r o m m u l t i v a r i a t e

n o r m a l i t y . S i m i l a r t o p r e v i o u s r e s e a r c h e r s ( e . g . ,

M u t h r n a n d K a p l a n , 1 9 8 5 , 1 9 9 2 ), w e f o u n d s ig -

n i f i ca n t p r o b l e m s a r i si n g w i t h u n i v a r i a t e s k e w n e s s

o f 2 .0 a n d k u r t o s e s o f 7 .0 . F u r t h e r r e s e a r c h i s

n e e d e d t o b e t t e r u n d e r s t a n d m o r e p r e c i se l y w h e n

n o n n o r m a l i t y b e c o m e s p r o b l e m a t i c , b u t i t s e e m s

c l e a r th a t o b t a i n e d u n i v a r i a te v a l u e s a p p r o a c h i n g

a t l e a s t 2 .0 a n d 7 . 0 f o r sk e w n e s s a n d k u r t o s e s a r e

s u s p e c t. S e c o n d , w e a g r e e w i t h p r e v i o u s r e s e a r c h -

e r s ( e . g ., H u e t a l ., 1 9 9 2 ; M u t h r n K a p l a n , 1 9 9 2 )

t h a t t h e A D F X 2 n o t b e u s e d w i t h s m a l l s a m p l e

s i z e s . A l t h o u g h w e f o u n d a d e q u a t e b e h a v i o r a t

s a m p l e s a s s m a l l a s N = 5 0 0 , o t h e r r e s e a r c h e r s

h a v e f o u n d p r o b l e m s w i t h t h e A D F X 2 a t s a m p l es

a s l a r g e a s N = 5 , 00 0 w h e n t e s t i n g m o r e c o m p l e x

m o d e l s ( H u e t a l. , 1 9 9 2) . T h e r e a r e s o m e e p i d e m i -

o l o g i c a l a n d c a t c h m e n t a r e a s t u d i e s t h a t d o h a v e

t h e s e l a r g e s a m p l e s i z e s a v a i l a b l e , a n d i n t h e s e

c a s es t h e A D F i s a p r o m i s in g m e t h o d o f e s t im a -

t i o n , p a r t i c u l a r l y f o r s m a l l e r m o d e l s . R e c e n t r e -

s e a r c h h a s a l s o s h o w n t h e p o s s i b i l i ty o f u s in g b o o t -

s t r ap p i n g t e c h n i q u e s to c o m p u t e m o r e s t a b l e A D F

5 Note tha t a l though the ob ta ined M L 2 values in-

c reased wi th inc reas ing nonnorm al i ty , the expec ted M L

X2 values we re e qual across dis tr ibut ion.

6 We thank b o th Albe r t S a tor ra and P e te r B ent le r ,

whom each independent ly sugges ted th i s s am e a rgu-

me nt as a potent ia l exp lanat ion for the ob tained resul ts .


12/14

C H I - S Q U A R E T E S T S T A T I S T IC S 2 7

X 2 e s t i m a t e s ( Y u n g a n d B e n t l e r , 1 9 94 ); h o w e v e r ,

m o r e w o r k i s n e e d e d t o e x p l o r e t h e u t il it y o f th i s

a p p r o a c h i n a p p l i e d r e s e a r c h s e t t i n g s .

F i n a l l y , r e l a t i v e t o t h e M L X 2 a n d t h e A D F X 2,

t h e S B X 2 b e h a v e d e x t r e m e l y w e l l i n n e a r l y e v e r y

c o n d i t i o n a c r o s s s a m p l e si z e , d i s t r i b u t i o n , a n d

m o d e l s p e c i f i c a ti o n . A d d i t i o n a l l y , t h e S B X 2 h a d

t h e d e s i r a b l e p r o p e r t y o f s im p l i f y in g t o t h e M L X 2

u n d e r m u l t i v a r i a t e n o r m a l i t y . W e t h u s r e c o m -

m e n d r e p o r t i n g b o t h t h e M L X 2 a n d t h e S B X 2

w h e n n o n n o r m a l d a t a i s s u s p e c t e d w i t h th e c l e a r

r e a l iz a t i o n t h a t t h e l o w e r S B v a l u e m a y b e r e -

f le c t in g d e c r e a s e d p o w e r a n d n o t s i m p l y t h a t th e

m o d e l i s a b e t t e r f it t o t h e d a t a b a s e d o n t h e S B

X 2. M o d e l f i t s h o u l d t h u s b e e v a l u a t e d w i t h a p p r o -

p r i a t e c a u t io n . T h e r e a r e a fe w d i s a d v a n t a g e s t o

u s i n g t h e S B X 2 i n p r a c t ic e . O n e i s t h a t t h e c o m p u -

t a t i o n o f t h e S B X 2 r e q u i r e s r a w d a t a , w h i c h m i g h t

p o s e a p r o b l e m f o r s o m e r e s e a r c h e r s . S e c o n d , t h e

S B X 2 i s c u r r e n t l y o n l y a v a i l a b l e i n E Q S . T h i s

p o s e s a p r a c ti c a l p r o b l e m f o r r e s e a r c h e r s w h o a r e

e i t h e r n o t t r a i n e d i n o r d o n o t h a v e a c c e s s t o E Q S .

e f e r e n c e s

Bent ler , P . M. (1989) . E QS: S t ruc t ura l equa t i ons pro-

gram manual . L os Ange l es : BM DP S t a t i s t i ca l

S of t ware .

Bol len, K. A. (1989) . Structural equations with latent

variables. New York : W i l ey .

Browne , M . W . (1982) . Cov ar i ance s t ruc t u res . I n D . M .

Hawki ns (E d . ) , Topics in appl ied mul t ivariate analysis

(pp . 72-141) . Cambr i dge : Cambr i dge Uni ve r s i t y

Press .

Brow ne , M . W . (1984). A sym pt o t i c d i s tr i bu t ion f r ee

me t hods i n t he ana l ys i s o f cova r i ance s t ruc t u res . Brit-

i sh Jour nal o f Mathem at ical and Stat is tical Psychol -

ogy, 37, 127-141.

Brow ne, M. W., Mels , G. , Cow ard, M. (1994). Path

Ana l ys i s: RA M ON A. In L . W i l k i nson M . A . H i l l

(Eds. ) , Systat : Ad va nce d appl icat ions (pp. 163-224) .

E v a n s t o n , I L : S Y S T A T .

Cho u, C. P . , Ben t ler , P . M. , Sato rra , A. (1991) . Scaled

t e s t s ta t is t ic s and rob us t s t anda rd e r ro r s f o r non-no r -

ma l da t a i n cova r i ance s t ruc t u r e ana l ys i s : A M ont e

Car l o s t udy . B r i ti sh Journa l o f M at hemat i ca l and S ta -

t ist ical Psychology, 44, 347-357 .

F l e i shman , A . I . ( 1978). A me t hod fo r s i mul a t ing non-

norm al d i s t ri bu t ions . Psychometr ika, 43, 521-532 .

Hu , L . , Ben t ler , P . M. , Ka no, Y. (1992). Ca n tes t

s t a ti s ti c s i n cova r i ance s t ruc t u r e ana l ys is be t r us t ed?

Psycholog ical Bulle tin , 112, 351-362 .

J /Sreskog, K. G. , SSrbo m, D. (1993). L I S R E L - V I I I :

A guide to the prog ram and appl icat ions . M ooresv i l l e ,

IN: Scientific Software.

Kapl an , D . ( 1988). T h e i mpac t o f spec i f ica t ion e r ro r on

t he e s t i ma t i on , t e s ti ng , and i m prov em ent o f s tr uc t u r a l

equa t i on mode l s . Mult ivariate Behavioral Research,

23, 69-86 .

Kapl an , D . ( 1989). A s t udy o f t he sam pl i ng va r i ab i li t y o f

t he z -va l ues o f pa r am et e r e s t i ma t es f rom m i sspec if i ed

s t ruc t u r a l equa t i on mode l s . Multivariate Behavioral

Research, 24, 41-57 .

Law ley, D. N. , Maxw el l , A. E . (1971). Facto r analysis

as a statist ical method. L ondon: But t e rwor t h .

MacCal lum, R. (1986) . Speci f icat ion searches in covar i -

ance s t ruc t u r e mode l i ng . Psychological Bulletin,

100, 107-120.

M acCa l l um, R . ( 1990) . T he need fo r a l t e rna t i ve mea-

sures o f f i t i n cova r i ance s t ruc t u r e mode l i ng . Mult ivar-

iate Behavioral Research, 25, 157-162.

M acCa l l um, R . C . , Rozno wski , M . , Necowi t z , L . B .

(1990) . Model modif icat ions in covar iance s t ructure

ana l ys i s : T he p rob l em of cap i t a l i za t i on on chance .

Psycholog ical Bulle tin , 111, 490-504 .

Miccer i , T . (1989) . The unicorn, the normal curve, and

o t he r i mprobab l e c r ea t u r es . Psychological Bulletin,

105, 156-166.

Mu th6n, B. (1987) . L IS CO M P : A nal ys i s o f l inear s truc-

t ura l equa t i ons w i t h a comprehens i ve measurement

mode l . Mooresvi l le , IN: Scient i f ic Sof tware .

M ut h6n , B ., Kap l an , D . ( 1985). A compa r i son o f som e

met hodol og i es fo r t he f ac t o r ana l ys i s o f non-normal

L i ke r t va r i ab l e s . B r i ti sh Journa l o f Mat hema t i ca l and

Statist ical Psychology, 38, 171-189.

M ut h6n , B ., Kap l an , D . ( 1992). A com par i son o f som e

met hodol og i es fo r t he f ac t o r ana l ys i s o f non-normal

L i ke r t va r i ab le s : A n o t e on t he s ize o f t he mode l .

Bri t i sh Journal of Mathemat ical and Stat i s t ical Psy-

chology, 45, 19-30 .

Sar is , W. E . , S t ronk hors t , H. (1984). Causal model l ing

in nonexperimental research. A m s t e r d a m : S o c i o m e t -

r i c Resea rch F ounda t i on .

SAS Inst i tute , Inc . (1990) . SA S/ S TA T user 's guide: Vols .

1 and 2. Cary , NC: Aut hor .

Satorra , A. (1990) . Robustness i ssues in s t ructural equa-

t i on mode l i ng : A r ev i ew of r ecen t deve l opme nt s .

Qu ality Quantity, 24, 367-386 .

Satorra , A. (1991, June) . A sympt o t i c robus t i n f e rences

in the analys is o f mea n and covariance s t ructure . B a r -

ce l ona , S pa i n : Uni ve r s i t a t P ompeu F abra .


13/14


S a t o r r a , A . , B e n t l e r , P . M . ( 1 9 8 8 ). S c a l i n g c o r r e c t i o n s

f o r c h i - s q u a r e s t a t i s ti c s in c o v a r i a n c e s t r u c t u r e a n a l y -

sis . A S A Proceedings o f the Business and Econom ic

Section 3 0 8 - 3 1 3 .

S a t o r r a , A . , S a r i s, W . E . ( 1 98 5 ). T h e p o w e r o f t h e

l i k e l i h o o d r a t i o t e s t i n c o v a r i a n c e s t r u c t u r e a n a l y s i s .

Psychometrika 50 8 3 - 9 0 .

V a l e , C . D . , M a u r e l l i , V . A . (1 9 8 3 ) . S i m u l a t i n g m u l t i -

v a r i a t e n o n n o r m a l d i s t r i b u t i o n s . Psychometrika 48

4 6 5 - 4 7 1 .

Y u n g , Y . - F . , B e n t l e r , P . M . ( 1 99 4 ). B o o t s t r a p - c o r -

r e c t e d A D F t e s t s t a ti s t ic s i n c o v a r i a n c e s t r u c t u r e a n a l -

ysis . British Journal of Mathematical and Statistical

Psychology 47 6 3 - 8 4 .

A p p e n d i x

T e c h n i c a l D e t a i l s

D a t a G e n e r a t io n

E Q S g e n e r a t e s t h e r a w d a t a w i t h n o n - z e r o s k e w n e s s

a n d k u r t o s i s u si n g th e f o r m u l a e d e v e l o p e d b y F l e i s h m a n

( 1 97 8 ) i n a c c o r d a n c e w i t h t h e p r o c e d u r e s d e s c r i b e d b y

V a l e a n d M a u r e l l i (1 9 83 ) . T h e r a w d a t a w e r e g e n e r a t e d

b a s e d u p o n t h e c o v a r i a n c e m a t r i x i m p l i e d b y t h e m o d e l

p a r a m e t e r s f o r e a c h o f t h e t h r e e m o d e l s . T h e a v a i l a b i l i t y

o f th e r a w d a t a w a s n e c e s s a r y f o r th e c o m p u t a t i o n o f

t h e A D F a n d S B X2 tes t s t a t i s t i c s .

T h e s a m p l e s w e r e c r e a t e d u s i n g t h e m o d e l i m p l i e d

p o p u l a t i o n c o v a r i an c e m a t r i x ~ ( 0) . T h e m e a s u r e m e n t

e q u a t i o n s c o n s i s t e d o f th e p o p u l a t i o n p a r a m e t e r v a l u e s

t h a t d e f i n e d t h e p a r t i c u l a r m o d e l . E Q S g e n e r a t e d t h e

p o p u l a t i o n c o v a r i a n c e m a t r i x b a s e d o n t h e s e m e a s u r e -

m e n t e q u a t i o n s . T h e s a m p l e r a w d a t a w e r e c r e a t e d u s i n g

a r a n d o m n u m b e r g e n e r a t o r i n c o n j u n c t i o n w i t h t h e

c h a r a c t e r i s t ic s o f t h e p o p u l a t i o n c o v a r i a n c e m a t r i x . T h e

r a w d a t a w e r e g e n e r a t e d u n d e r t w o c o n s t r a i n t s : ( a ) t h e

e x p e c t e d v a l u e o f S s h o u l d e q u a l t h e p o p u l a t i o n c o v a r i -

a n c e m a t r i x ~ ( 0 ) , a n d ( b ) t h e e x p e c t e d v a l u e o f t h e

i n d i c e s o f s k e w n e s s a n d k u r t o s i s s h o u l d e q u a l t h e v a l u e s

s p e c i fi e d f o r e a c h m e a s u r e d v a r i a b l e .

V e r i f i c a ti o n o f D a t a G e n e r a t i o n

T o v e r i f y t h a t E Q S p r o p e r l y g e n e r a t e d t h e r a w d a t a

i n a c c o r d a n c e w i t h t h e d e s i r e d l e v e l s o f s k e w n e s s a n d

k u r t o s e s , t h r e e s e t s o f ra w d a t a o f s a m p l e s i z e N =

6 0 , 0 0 0 w e r e g e n e r a t e d . T h e t h r e e d a t a s e t s w e r e p r o -

d u c e d u s i n g t h e s a m e p r o c e d u r e s t h a t c r e a t e d t h e m u l t i -

v a r i a t e n o r m a l , m o d e r a t e l y n o n n o r m a l , a n d s e v e r e l y

n o n n o r m a l d i s t r i b u t i o n s f o r t h e s i m u l a t i o n s . T h e l a r g e

s a m p l e s i z e p r o v i d e s a m o r e a c c u r a t e e s t i m a t e o f th e

c o e f f ic i e n ts o f s k e w n e s s a n d k u r t o s i s f o r t h e g e n e r -

a t e d d a t a .

U n i v a r i a t e s k e w n e s s a n d k u r t o s e s w e r e c o m p u t e d f o r

t h e t h r e e s a m p l e s o f N = 6 0 ,0 0 0 u s i n g S A S P R O C

U N I V A R I A T E . F o r t h e n o r m a l l y d i s t ri b u t e d c o n d i t io n

( s k e w n e s s = 0 , k u r t o s i s = 0 ) , t h e m e a n u n i v a r i a t e sk e w -

n e s s f o r th e n i n e v a r i a b l e s w a s . 00 1 a n d t h e m e a n k u r t o -

s is w a s . 0 04 . F o r t h e m o d e r a t e l y n o n n o r m a l d i s t r i b u t i o n

( s k e w n e s s = 2 .0 , k u r t o s i s = 7 .0 ) , t h e m e a n s k e w n e s s

w a s 1 .97 3 a n d t h e m e a n k u r t o s i s w a s 6 .64 8. F i n a l l y , f o r

t h e s e v e r e l y n o n n o r m a l d i s t r i b u t i o n (s k e w n e s s = 3 .0 ,

k u r t o s i s =

curran,west&finch (1996)

Documents