logic and purpose significance testing

8/13/2019 Logic and Purpose Significance Testing

1/12

Psychological Methods Copyright 1997 by the American Psychological Association, Inc.1997, Vol. 2, No. 2, 161-172 1082-989X/97/ 3.00

On the og ic and Purpose o f S ignif icance Test ingJose M. Cor t ina

George Mason UniversityWil l i am P. Dun lap

Tulane UniversityThere has been much recent attention given to the problems involved with thetraditional approach to null hypothesis significance testing (NHST). Many havesuggested that, perhaps, NHST should be abandoned altogether in favor of otherbases for conclusions such as confidence intervals and effect size estimates (e.g.,Schmidt, 1996). The purposes of this article are to (a) review the function that dataanalysis is supposed to serve in the social sciences, (b) examine the ways in whichthese functions are performed by NHST, (c) examine the case against NHST, and(d) evaluate interval-based estimation as an alternative to NHST.

The topic of this article is null hypothesis signifi-cance testing (NHST; Cohen, 1994). By this we meanthe process, common to the behavioral sciences, ofrejecting or suspending judgment on a given null hy-pothesis based on a priori theoretical considerationsand p values in an attempt to draw conclusions withrespect to an alternative hypothesis. We should beginby saying that we agree with J. Cohen, G. Gigerenzer,D. Bakan, W. Rozeboom, and so on with respect tothe notion that the logic of NHST is widely misun-derstood and that the conclusions drawn from suchtests are often unfounded or at least exaggerated(Bakan, 1966; Cohen, 1990, 1994; Gigerenzer, 1993;Rozeboom, 1960). Nevertheless, we think it importantthat the extent and likely direction of such problemsbe carefully examined, because NHST, when usedand interpreted properly, is useful for certain purposesand is only partially problematic for others.

This article is divided into three parts. The first partis devoted to an examination of the role played bydata analysis in the social sciences and the extent to

Jose M. Cortina, Department of Psychology, George Ma-son University; William P. Dunlap, Department of Psychol-ogy, Tulane University.We are most grateful for the comments of John Hollen-beck, Richard P. DeShon, Mickey Quinones, Sherry Ben-nett, Frank Schmidt, Jack Hunter, and the members of theBehavioral Science Research Group, without whom this ar-ticle would have suffered.Correspondence regarding this article should be ad-dressed to Jose M. Cortina, Department of Psychology,MSN 3F5, 4400 University Drive, Fairfax, Virginia 22030.Electronic mail may be sent via Internet to [email protected].

which NHST supports this role. The second partpresents and critiques the case against NHST as ananalytic tool. The third part investigates the viabilityof confidence intervals as an alternative to NHST.

Purpose of Data AnalysisPerhaps the best way to begin is to consider the

rationale of the typical research project in the socialsciences and the role played by data analysis in thestructure of the project. Every research project beginswith a research question. Let us assume that the ques-tion has to do with a bivariate relationship that hasnever been examined empirically before. The firststep in the project typically involves the generation ofa theoretically based answer to the question, that is, ahypothesis. This hypothesis is often based on a com-bination of reason and previous empirical work inrelated areas and gives us evidence of a certain kindfor a particular answer to the research question. Thisform of evidence is invaluable, but scientific traditionholds that corroboration, whether in the form of fal-sification or justification, is desirable (Lakatos, 1978;Popper, 1959; Serlin Lapsley, 1985). If that cor-roboration is independent of the theoretical basis forthe initial answer to the research question, then thecorroboration is all the more impressive. This inde-pendent corroboration often takes the form of empiri-cal data and allows us to attack the research questionfrom directions that are largely orthogonal to one an-other.

The stronger the theoretical basis for the initial an-swer, the less reliance one need put on the data. Forexample, it is said that Albert Einstein had no interestin empirical tests of his theory of relativity. The

161


2/12

1 6 2 C O R T I N A A N D D U N L A Pt h e o r y w a s s o s t r o n g t h a t a n y c o m p e t e n t l y c o l l e c t e de m p i r i c a l e v i d e n c e w o u l d s u p p o r t t h e t h e o r y . T h u s ,t h e e m p i r i c a l e v i d e n c e w a s l a r g e l y s u p e r f l u o u s .

O n t h e o t h e r h a n d , w e i n t h e s o c i a l s c i e n c e s a r er a r e l y i f e v e r ju s t i f i e d i n p l a c i n g s o m u c h c o n f i d e n c ei n th e o r e t i c a l a n s w e r s . N e v e r t h e l e s s , w e c a n c e r t a i n l yt h in k o f s u c h h y p o t h e s e s a s l y i n g o n a c o n t i n u u m o fs u p p o r t a b i l i ty t h a t r a n g e s f r o m w e a k t o s t r o n g . I f m u l -t i p l e , w e l l - r e s p e c t e d t h e o r e t i c a l p e r s p e c t i v e s s u g g e s tt h e s a m e h y p o t h e s i s , t h e n t h e t h e o r e t i c a l s u p p o r t f o rt ha t pos i t i on i s s t r ong , a nd one i s l e s s r e l i a n t on t hed a t a . I f, o n t h e o t h e r h a n d , t h e t h e o r e t i c a l s u p p o r t f o ra g i v e n p o s i t i o n i s w e a k , t h e n t h e b u r d e n o f p r o o fs h i f t s t o t he da t a .

L e t u s a s s u m e f o r t h e m o m e n t t h a t t h e t h e o r e t i c a lb a s i s f o r a g i v e n h y p o t h e s i s i s o f a v e r a g e s t r e n g t h . L e tu s a s s u m e f u r t h e r t h a t t h e e x p e r i m e n t d e s i g n e d t op r o v i d e c o r r o b o r a t i o n f o r t h i s h y p o t h e s i s i n v o l v e ss o u n d m e a s u r e s , r e a s o n a b l e p r o c e d u r e s , a n d s o o n ,a n d t h a t d a t a a r e c o l l e c t e d w i t h i n t h i s d e s i g n . T h eN e y m a n - P e a r s o n f r a m e w o r k o n w h i c h m u c h o f m o d -e m s i g n i f i c a n c e t e s t i n g i s b a s e d s u g g e s t s t h a t t h e d e -s i r e d o u t c o m e o f d a t a a n a l y s i s is t h e a d j u s t m e n t o f o u rc o n f i d e n c e i n o u r h y p o t h e s e s s o t h a t w e c a n b e h a v e a si f a g i v e n h y p o t h e s i s i s t ru e o r f a l s e u n t i l f u r t h e r e v i -d e n c e i s a m a s s e d ( N e y m a n & P e a r s o n , 1 9 2 8 , 1 9 3 3 ) .B u t t h e q u e s t i o n r e m a i n s , h o w d o e s o n e d e c i d ew h e t h e r a n d t h e e x t e n t t o w h i c h t h e d a t a a l l o w u s t oi n c r e a s e o u r c o n f i d e n c e i n t h e t h e o r e t i c a l l y b a s e d a n -s w e r t o t h e r e s e a r c h q u e s t i o n ?

L e t u s c o n s i d e r f o r a m o m e n t w h a t i t i s t h a t w eh a v e t o w o r k w i t h i n t h e s o c i a l s c i e n c e s , p a r t i c u l a r l yi n c o m p a r i s o n w i t h t h e p h y s i c a l s c i e n ce s . L a r g e l y b e -c a u s e o f th e c o m p l e x i t y o f t h e v a r i ab l e s s t u d i e d i n t h es o c i a l s c i e n c e s , o u r t h e o r i e s a r e n o t p o w e r f u l e n o u g ht o g e n e r a t e p o i n t h y p o t h e s e s . P h y s i c a l s c i e n t i st s o f t e ns c o f f a t o u r a t t e m p t s t o m a k e g e n e r a l s t a t e m e n t s o f t h ef o r m A i n c r e a s es B w i t h o u t s p e c i f y in g t h e p r e c is ed e g r e e t o w h i c h A i n c r e a s e s B . T h e e x a m p l e o f 3 2f t . / s / s h a s b e e n u s e d i n a r g u i n g t h a t w e n e e d t o f o c u so n p a r a m e t e r e s t i m a t i o n ( e . g . , r e g r e s s i o n w e i g h t s ) i n -s t ea d o f s a y i n g t h i n g s s u c h as G r a v i t y m a k e s t h i n g sf a i l . A l t h o u g h w e a g r e e w i t h t h is s e n t i m e n t i n p r i n -c i p l e , it s tr i k e s u s a s b e i n g a b i t n a i v e w h e n a p p l i e d t ot h e s t u d y o f h u m a n b e h a v i o r . T h e c o n s t ra i n ts p l a c e do n p o p u l a t i o n v a l u e s b y t h e o r y a r e m u c h w e a k e r i nt h e s o c i a l s c i e n c e s t h a n t h e y a r e i n t h e p h y s i c a l s c i -e n c e s ( S e r l in & L a p s l e y , 1 9 8 5 ) . T h e e x a m p l e o f 3 2f t ./ s /s i s a m o r e o r l e s s i m m u t a b l e l a w o f n a t u r e ( o nE a r t h o f c o u r s e ) . I t i s a l w a y s t r u e , s o i t m a k e s s e n s e t oa p p l y a s p e c i f i c n u m b e r t o i t . W h a t h a p p e n s i f w e

c h a n g e t h e q u e s t io n a b i t? S u p p o s e w e w i s h t o k n o wt h e r a t e a t w h i c h s t o c k p r i c e s d r o p a f t e r in t e r e s t r a te sa r e r a is e d . I n s o m e c a s e s , a g i v e n i n c r e a s e i n i n t e re s tr a t e s p r o d u c e s a p r e c i p i t o u s d r o p ; i n o t h e r c a s e s t h es a m e i n c r e a s e p r o d u c e s l i t tl e o r n o d r o p . W e c a n c o m -p u t e a n a v e r a g e a n d u s e t h i s t o d r a w s o m e g e n e r a lc o n c l u s i o n s , b u t i t w o u l d b e d i f f i c u l t t o p r e d i c t t h ed r o p r a t e i n a n y g i v e n i n s t a n c e w i t h a g r e a t d e a l o fa c c u r a c y . T h e p o i n t i s t h a t t h e v a g a r i e s o f t h e s t o c km a r k e t a n d o t h e r s u c h p h e n o m e n a a r e c r e a t e d b y h u -m a n d e c i si o n s a n d b e h a v i o r . T h e r e i s o f t en t o o m u c hc o m p l e x i t y i n s u c h s i t u a t i o n s a n d p e r h a p s t o o m u c hv a r i a b i l i t y i n p e o p l e f o r a t h e o r e t i c a l l y b a s e d p o i n tp a r a m e t e r e s t im a t e t o m a k e m u c h s e ns e . W e s u g g e s tt h a t t h e r e a r e m a n y s i t u a t i o n s i n w h i c h w e a r e b e t t e ro f f s t i c k i n g t o m o r e c o n s e r v a t i v e , g e n e r a l p r e d i c t i o n s( a n d c o n c l u s i o n s ) s u c h a s A i n c r e a s e s B .

R e g a r d l e s s o f t h e f o r m o f t h e h y p o t h e s is , i t m u s t b ee x c l u s i o n a r y i n o r d e r t o m a k e a c o n t r i b u t i o n . T h i su s u a l l y , t h o u g h b y n o m e a n s n e c e s s a r i l y , t a k e s t h ef o r m o f a d i c h o t o m o u s p r e d i c t i o n o f s o m e k i n d . F o re x a m p l e , t h e h y p o t h e s e s A a f f ec t s B a n d T h e9 5 % c o n f i d e n c e in t e r v a l w i l l n o t c o n t a i n z e r o a r ed i c h o t o m o u s a n d e x c l u s i o n a r y i n t h e s e n s e t h a t Ae i t h e r a f f e c t s B o r d o e s n o t , o r t h e i n t e r v a l c o n t a i n sz e r o o r n o t , a n d o u r h y p o t h e s e s e x c l u d e t h e a l t e r n a -t i v e p o s s ib i l it i e s. E m p i r i c a l e v i d e n c e i s t h e n g a t h e r e da n d e v a l u a t e d i n t e r m s o f t h e e x te n t t o w h i c h w e c a na d j u s t o u r c o n f i d e n c e i n t h e r e l e v a n t h y p o t h e s e s . I no t h e r w o r d s , w e u s e e m p i r i c a l c o r r o b o r a t i o n ( o r l a c kt h e r e o f ) t o a d j u s t a d e g r e e o f r a t i o n a l b e l i e f ( K e y n e s ,1 9 2 1 ) . W h e r e i t is n e c e s s a r y t o m a k e a d e c i s i o n b a s e do n t h e s e l e v e l s o f c o n f i d e n c e , w e t h e n b e h a v e a s i f t h en u l l w e r e t r u e o r w e b e h a v e a s i f t h e a l t e r n a t iv e w e r et r ue . T h i s b e h a v i o r c a n t a k e t h e f o r m o f p o l i c yc h a n g e , a s i s o f t e n t h e c a s e i n t h e a p p l i e d s o c i a l s c i -e n c e s , o r i t c a n r e p r e s e n t o n e s t e p i n a s e r ie s o f s t e p sa s s o c i a t e d w i t h t h e t e s t o f a t h e o r y ( e .g . , p a t h a n a l y s i so r m e a s u r e m e n t d e v e l o p m e n t ) . A s a l w a y s , it b e -h o o v e s u s t o k e e p i n m i n d t h a t t h e u n d e r l y i n g b a s i sf o r t h e d e c i s i o n i s c o n t i n u o u s .

W h a t i s r e q u i r e d o f e m p i r i c a l c o r r o b o r a t i o n ?T h e r e a r e a t l e a s t t h r e e i n t e r r e l a t e d r e q u i r e m e n t s f o re m p i r i c a l c o r r o b o r a t i o n : ( a ) o b j e c t i v i t y , ( b ) e x c l u s i o no f a l t e rn a t i v e h y p o t h e s e s ( e .g . , a h y p o t h e s i s o f a re -l a t i ons h i p w i t h a d i f f e r e n t s i gn o r a n a l t e r na t i ve s t r uc -t u r a l m o d e l ) , a n d ( c ) e x c l u s i o n o f a l t e rn a t i v e e x p l a -n a t i o n s ( e . g . , c o n f o u n d s o r s a m p l i n g e r r o r ) . I t c a n b es h o w n t h a t p r o p e r e x p e r i m e n t a l d e s i g n f o l l o w e d b ys i g n i f i c a n c e t e s t i n g a l l o w s o n e t o a d d r e s s a l l t h r e e o ft h e s e r e q u i r e m e n t s . N o a n a l y s i s p r o c e d u r e , i n c l u d i n g


3/12

L O G I C A N D T H E N U L L 1 6 3s i g n i f ic a n c e t e s t in g , c a n m e e t t h e s e r e q u i r e m e n t s p e r -f ec t l y , bu t s i gn i f i cance t e s t i ng , f o r a va r i e t y o f r ea -sons , addr es se s t hese c r i t i ca l i s sues a s we l l a s o r be t t e rt h a n d o t h e a l t e rn a t i v e s f o r m a n y t y p e s o f r e se a r c hques t i ons . Be f or e d i scus s i ng t hese t h r ee r equ i r ement s ,h o w e v e r , l e t u s t a k e a m o m e n t t o c o n s i d e r th e r e a s o nt ha t co r r obor a t i on i t s e l f i s c r i t i ca l .

I n t h e p e r f e c t w o r l d , p e r h a p s e v e r y p i e c e o f r e -s e a r c h w o u l d b e p u b l i s h e d s o m e w h e r e . I n a s l i g h t l yl e s s p e r f e c t w o r l d , a n y s t u d y w i t h c o m p e l l i n g t h e o r ya n d a n a d e q u a t e m e t h o d w o u l d b e p u b l i s h e d s o m e -w h e r e . G i v e n l i m i te d jo u r n a l s p a c e , h o w e v e r , w e a r ef o r c e d t o c h o o s e f r o m a m o n g s u b m i s s i o n s t h o s e a r -t i c l e s t ha t advance knowl edge w i t h t he g r ea t e s t e f f i -c i ency . T he a r t i c l e t ha t p r ov i des a ce r t a i n t heor e t i ca lanswer t o a ques t i on , bu t a l so p r esen t s da t a t ha t sug-g e s t a d i f f e r e n t a n s w e r , d o e s a d v a n c e k n o w l e d g e .H o w e v e r , a l l e l s e b e i n g e q u a l , i t d o e s n o t a d v a n c eknowl edge a s f a r a s does t he a r t i c l e w i t h t heor e t i ca la n d e m p i r i c a l a n s w e r s t h a t a r e in a g r e e m e n t . T h e f i rs tt y p e i n f o r m s u s t h a t e i th e r t h e t h e o r y i s w r o n g o r t h em e t h o d i s f l a w e d . T h u s , w e a r e u n a b l e t o a d j u s t o u rc o n f i d e n c e i n a n y p a r t i c u l a r a n s w e r t o t h e r e l e v a n tr e sea r ch ques t i on . T he second t ype , by con t r a s t , a l -l o w s u s t o i n c r e a s e o u r c o n f i d e n c e i n a g i v e n a n s w e rb y v i r t u e o f t h e i n d e p e n d e n t s o u r c e s s u g g e s t i n g t h a tanswer . G i ven t ha t co r r obor a t i on i s i mpor t an t , s i gn i f i -c a n c e t e s t i n g i s u s e f u l b e c a u s e i t g i v e s u s a m e c h a -n i s m f o r a d j u s ti n g o u r c o n f i d e n c e i n c e r ta i n a n s w e r s .T h i s c a n b e s h o w n i n t h e c o n t e x t o f t h e t h re e r e q u i r e -m e n t s f o r c o r r o b o r a t i o n m e n t i o n e d a b o v e .

O n e o f t h e m o s t c r it i ca l r e q u i r e m e n t s o f c o r r o b o -r a t i on i s ob j ec t i v i t y . T he des i r e f o r ob j ec t i ve ve r i f i ca -t i o n c a n b e t r a c e d a t l e a s t a s f a r b a c k a s K a n t ' s ( 1 7 8 1 )ritique of Pure Reason and h i s r e f e r ences t o i n t e r -s u b j e c t iv i t y , b u t t h e m o d e r n n o t i o n s o f o b j e c t i v i ty a n dv a l u e f r e e d o m i n th e s o c i a l s c ie n c e s a p p e a r to s t e mf r o m M . W e b e r ( M i l le r , 1 9 7 9 ). E m p i r i c a l c o r r o b o r a -t i on shou l d be ob j ec t i ve i n t he s ense t ha t i t shou l d bea s i n d e p e n d e n t o f t h e t h e o r e t ic a l l y b a s e d a n s w e r t ot he r e sea r ch ques t i on a s i s pos s i b l e ( De l l ow, 1970 ;M i l l e r , 1987) . I n th i s way , i t can co mp l em ent en t i r e l yt h e t h e o r e t i c a l e v i d e n c e . W h i l e p e r f e c t o b j e c t i v i t ym ay no t be poss i b l e ( Kuhn , 1962; L ak a t os , 1978) , i t i si m p o r t a n t t h a t w e s t r iv e f o r s e p a r a t io n o f t h e o r e t ic a la n d e m p i r i c a l e v i d e n c e .

I t i s l a r ge l y because o f t h i s need t o s epa r a t e t heo-r e t i c a l f r o m e m p i r i c a l e v i d e n c e t h a t B a y e s i a n s t a t i s -t i c s , w i t h i t s r e l i ance on a r b i t r a r y p r i o r p r obab i l i t i e s ,h a s b e e n u s e d s e l d o m t o t e s t s c i e n t i f i c h y p o t h e s e s( P u t nam, 1981) . I ns t ead , we a sk t ha t t heor e t i ca l an -

s w e r s , w h i c h a r e t y p i c a l l y d e v e l o p e d b y t h e r e -s e a r c h e r t h a t a s k e d t h e r e s e a r c h q u e s t i o n a n d a r et he r e f o r e suscep t i b l e t o i n t r a sub j ec t i v i t y , be s epa r a t eda s m u c h a s p o s s i b l e f r o m t h o s e e m p i r i c a l an s w e r s . I np r o p e r s i g n i f i c a n c e t e s t i n g w i t h a d e q u a t e l e v e l s o fp o w e r , w e c o m p a r e o b s e r v e d r e s u lt s w i t h a p r i o ri c u t -o f f s t o d e c id e w h e t h e r o r n o t w e w i l l a d j u s t o u r c o n -f i dence i n t he hypo t hes i s a t hand . T hese cu t o f f s a r es o m e w h a t a r b i t ra r y , b u t t h e y a r e a l s o i n t e rs u b j e c t iv ei n t he s ense t ha t t hey a r e conven t i ona l , and t hey a r ec h o s e n b e f o r e t h e d a t a a r e e x a m i n e d . T h u s , w h i l e i tc a n b e a r g u e d t h a t t h e c h o i c e o f c u t o f f is s o m e w h a tw h i m s i c a l , i t i s l a r g e l y t h e w h i m o f o t h e r s ( a s o p -p o s e d t o t h e e x p e r i m e n t e r a t h a n d ) t h a t d e t e r m i n e s t h ec u t o f f , a n d w h a t e v e r i m p a c t t h e w h i m o f t h e e x p e r i -m e n t e r h a s o n t h e c h o i c e o f c u t o f f , th i s i m p a c t t a k e sp l a c e p r i o r t o e x a m i n a t i o n o f d a ta . T h i s a l l o w s o n e t om e e t t h e P o p p e r i a n r e q u i r e m e n t t h a t t h e c o n d i t i o n sn e c e s s a r y f o r t h e d r a w i n g o f c e r t a in c o n c l u s i o n s a r ed e t e r m i n e d b e f o r e h a n d , c e t e r i s p a r i b u s c l a u s e s n o t -w i t hs t and i ng ( S e r l i n & L aps l ey , 1985) . O f cour se , ob-s e r v e d p r o b a b i l i t y v a l u e s s h o u l d b e r e p o r t e d s o t h a tt h e r e a d e r c a n d r a w h i s o r h e r o w n c o n c l u s i o n s , b u tt h e c o n c l u s i o n s o f th e e x p e r i m e n t e r a r e to b e b a s e d o npr ede t e r mi ned c r i t e r i a .

A s e c o n d d e m a n d o f e m p i r i c a l c o r r o b o r a t i o n is t h a ti t a ll ows us t o r u l e ou t a l t e r na t i ve hypo t he ses . I f ourp r e d i c t io n is t h a t A h a s a p o s i t i v e e f f e c t o n B , t h e ni t i s des i r ab l e t ha t our da t a a l l ow us t o a s ses s t hep l a u s i b il i ty o f a l t e rn a t i v e h y p o t h e s e s s u c h a s A h a sa n e g a t i v e e f f e c t o n B o r A h a s a n e g l i g i b le e f f e c to n B . I n t h is w a y , w e c a n s h o w th a t o u r th e o r ye x p l a i n s p h e n o m e n a t o a g r e a t e r e x t e n t t h a n d o a l t e r -na t i ves ( L aka t os , 1978) . S i gn i f i cance t e s t i ng a l l owst h e c o m p a r i s o n o f t h e o r et i c a l h y p o t h e s e s t o a n u l lh y p o t h e s i s . T h e t e r m null h a s t y p i c a l l y b e e n u s e d t or epr esen t t he hypo t he s i s t ha t i s t o be nu l l i f ied ( C ohen ,1994) . T h i s nu l l hypo t hes i s can be t ha t t he r e i s noe f f ec t o r no r e l a t i onsh i p , bu t t h i s need no t be t he case .Reg ar d l e s s o f the f o r m o f the nu l l , s i gn i f i cance t e s ti ngi n v o l v e s a c o m p a r i s o n o f h y p o t h e t i c a l d i s t r i b u t i o n s ;s p e c i f ic a l ly , i t i n v o l v e s t h e c o m p a r i s o n o f t h e s a m -p i i ng d i s t r i bu t i on a s soc i a t ed w i t h t he obse r ved r e su l tt o t he d i s t r i bu t i on a s soc i a t ed w i t h t he nu l l . I f it i sh i gh l y un l i ke l y t ha t t he nu l l d i s t r i bu t i on woul d havep r o d u c e d t h e o b s e r v e d r e s u l t, a n d i f th i s d i s c r e p a n c yi s i n t he an t i c i pa t ed d i r ec t i on , t hen we ad j us t our con-f i dences such t ha t we t en t a t i ve l y r u l e ou t t he nu l lhypo t hes i s ( a s we l l a s d i s t r i bu t i ons t ha t a r e even l e s spr obab l e t han t he nu l l ) i n f avor o f t he t heor e t i ca l l yb a s e d h y p o t h e s i s .


4/12

1 6 4 C O R T I N A A N D D U N L A PA t h i r d d e m a n d o f e m p i r i c a l c o r r o b o r a t i o n i s t h a t it

b e s u b j e c t t o a s f e w a l t e r n a t i v e e x p l a n a t i o n s a s p o s -s i bl e . T h u s , w e h o p e t o b e a b l e t o p o i n t t o o u r d a t a a n dc l a i m n o t o n l y t h a t it i s a s o n e w o u l d e x p e c t g i v e n t h et h e o r y , b u t a l s o t h a t i t s a l i g n m e n t w i t h t h e t h e o r y i sn o t d u e t o e x t r a n e o u s f a c t o r s . I f o n e a c c e p t s t h e a p -p r o a c h o f J o h n S t u a r t M i l l ( 1 8 7 2 ) , t h e n t h e r e a r e t h r e ec r i te r i a f o r th e i n f e r e n c e o f c a u s e : t e m p o r a l p r e c e -d e n c e , c o v a r i a t i o n , a n d t h e e l i m i n a t i o n o f a l te r n a t i v ee x p l a n a t i o n s f o r t h e c o v a r i a t io n . T h e f i r st t w o p r e s e n tr e l a t i v e l y f e w p r o b l e m s . I t i s t h e p r o b l e m o f a l t e rn a -t i v e e x p l a n a t i o n s t h a t g i v e s u s t h e m o s t t r o u b l e . F o ra n y g i v e n i n s t a n c e o f c o v a r i a t io n , t h e r e a r e l i k e l y t ob e a m u l t i t u d e o f p o s s i b l e a l t e r n a t i v e e x p l a n a t i o n ss u c h a s th e i m p a c t o f u n m e a s u r e d v a r i a b l e s, t h ec h o i c e o f s a m p l e , a n d s o o n . O n e p o t e n t i a l a l t e r n a ti v es o u r c e o f c o v a r i a t i o n t h a t h a s r e c e i v e d a g r e a t d e a l o fa t t e n t io n i n r e c e n t y e a r s i s s a m p l i n g e r r o r. 1 T h e pv a l u e i n N H S T g i v e s t h e p r o b a b i l i t y t h a t t h e d a t aw o u l d h a v e o c c u r r e d g i v e n t h e t r u th o f th e n u l l h y -p o t h e s i s ( C a r v e r , 1 9 7 8 ). I n o t h e r w o r d s , p i s th e p r o b -a b i l i ty t h a t t h e d e p a r t u r e o f t h e t e s t r e s u lt f r o m n u l lw o u l d h a v e r e s u l t e d f r o m s a m p l i n g e r r o r a l o n e . 2 A l -t h o u g h o u r c u t o f f s ( . 0 5 a n d . 0 1 t y p i c a l l y ) a r e a r b i -t r a r y , t h e y a l l o w u s t o e v a l u a t e t h e e x t e n t t o w h i c hs a m p l i n g e r r o r i s a v i a b l e a l t e r n a t i v e e x p l a n a t i o n . I fo u r p v a l u e i s l e s s t h a n t h e p r e s e t c u t o f f , t h e n w e c a nb e r e a s o n a b l y c o n f i d e n t ( a s s u m i n g a l l e l s e is i n o rd e r )t h a t s a m p l i n g e r r o r w a s n o t t h e r e a s o n t h a t o u r t e s ts t a t i s t i c d i f f e r e d f r o m o u r n u l l v a l u e . T h u s , w e h a v em o v e d t o w a r d s a t i s f y i n g t h e th i r d o f M i l l ' s c r i t e ri a f o ri n f e r r i n g c a u s a l i t y . W e h a v e e f f e c t i v e l y r u l e d o u ts a m p l i n g e r r o r a s a n a lt e r n a t iv e e x p l a n a t i o n f o r th ed e p a r t u r e o f o u r t e s t r e s u l t f r o m n u l l .

I n s u m m a r y , t h e p u r p o s e o f d a t a a n a l y s i s is t o p r o -v i d e c o r r o b o r a t i o n ( o r f a i l t o p r o v i d e c o r r o b o r a t i o n )o f t h e o r e t ic a l a n s w e r s t o r e s e a r c h q u e s t i o n s . T h i s c o r -r o b o r a t i o n i s m o s t c o n v i n c i n g w h e n i t i s i n t e r s u b j e c -t i v e a n d d i s c o n f i r m i n g o f a l te r n a t i v e h y p o t h e s e s a n de x p l a n at i o ns . W e c l e a r l y s h o w t h a t N H S T d o e s i n f a ctp r o v i d e a f o r m o f c o r r o b o r a t io n t h a t c o n t a in s e a c h o ft h e s e e l e m e n t s .

C a s e A g a i n s t N H S TT h e c a s e a g a i n s t N H S T h a s t a k e n v a r i o u s f o r m s .

T h e m o s t c o m m o n a t t ac k o f re c e n t y e a r s ha s i n v o l v e dp o i n t i n g o u t t h e i n t e r p r e t a t i o n a l p r o b l e m s a s s o c i a t e dw i t h N H S T w h e n c o n d u c t e d i n th e p r e se n c e o f m e t h -o d o l o g i c a l f la w s ( e . g ., s m a l l s a m p l e s ; S c h m i d t , 1 9 9 6 )a n d e x p e r i m e n t e r i g n o r a n c e ( e . g . , i n t e r p r e t a t i o n o f

c o n d i t i o n a l p r o b a b i l i t i e s o f e m p i r i c a l r e s u l t s a s c o n -d i t i o n a l p r o b a b i l i t ie s o f h y p o t h e s e s ; C o h e n , 1 9 9 4 ) .T h e r e c a n b e n o d o u b t t h a t t he p r e s e n c e o f m e t h o d -o l o g i c a l f l a w s i n a s t u d y l i m i t s t h e c o n c l u s i o n s t h a tc a n b e d r a w n f r o m N H S T . O f c o u r se , s u c h fl a w s l i m i tt h e c o n c l u s i o n s t h a t c a n b e d r a w n f r o m a n y p r o c e -d u r e , s o w e s e e n o p o i n t i n h o l d i n g N H S T o r a n yo t h e r p r o c e d u r e a c c o u n t a b l e f o r th e s h o r t c o m i n g s o ft h e d a t a t o w h i c h t h e y a r e a p p l i e d .

O n t h e o t h e r h a n d , w e w o u l d l i k e t o a d d r e s s w h a tw e c o n s i d e r t o b e t h e m o s t g l a r i n g p r o b l e m s a s s o c i -a t e d w i t h th e c r i ti c i s m s i n v o l v i n g e x p e r i m e n t e r i g n o -r a n c e . T h e c r i t i c i s m s a r e t h a t ( a ) m a n y e x p e r i m e n t e r sc o m m i t t h e e r r o r o f i n t e r p r e ti n g t h e c o n d i t i o n a l p r o b -a b i l it y o f t h e e m p i r i c a l r e s u lt , P ( D I / -/ ) , a ls o k n o w n a st h e p v a l u e , a s t h e c o n d i t i o n a l p r o b a b i l i t y o f t h e h y -p o t h e s i s , P ( H I D ) ; ( b ) th e p r o b a b i l i s t i c n a t u r e o fN H S T c r e a te s l o g i c a l p r o b l e m s ; a n d ( c ) N H S T i s m i s -l e a d i n g i n t h a t i t f o c u s e s o n c o n t r o l o f T y p e I e r r o r sw h e n t h e p r o b a b i l i t y o f s u c h e r r o r s i s z e r o .Interpretation of p

T h e p v a l u e g i v e s t h e p r o b a b i l i t y t h a t t h e o b s e r v e de m p i r i c a l r e s u l t w o u l d h a v e o c c u r r e d g i v e n a c e r t a i nh y p o t h e t i c a l d i s t r i b u t i o n . W e a g r e e c o m p l e t e l y w i t hC o h e n ( 1 9 9 4 ), G i g e r e n z e r ( 1 9 9 3 ), R o z e b o o m ( 1 9 60 ) ,a n d o t h e r s i n t h e i r o b s e r v a t i o n t h a t m a n y r e s e a r c h e r st e n d t o m i s i n t e r p r e t t h e p v a l u e a s t h e p r o b a b i l i t y o f ah y p o t h e s i s g i v e n t h e d a t a . H o w e v e r , t h is i s th e f a u l t o ft h o s e w h o a r e d o i n g t h e i n t e r p r e ti n g , n o t t h e t o o l s th a tt h e y c h o o s e . A l s o , i t i s i m p o r t a n t t o r e c o g n i z e t h a tt h e r e a r e m a n y s i t u a t i o n s i n w h i c h i n t e r p r e ta t i o n o f as m a l l v a l u e o f P DIHo) a s i n d i c a t i n g a s m a l l v a l u e o fP ( H o l D ) c a n m a k e s o u n d , p r a c t i c a l s e n s e . T o s h o wt h i s , l e t u s f i r s t e x a m i n e C o h e n ' s ( 1 9 9 4 ) e x c e l l e n te x a m p l e f r o m c l i n i c a l / a b n o r m a l p s y c h o l o g y . I t g o e sl ike th is .

T h e b a s e r a t e f o r s c h i z o p h r e n i a i n a d u l t s i s r o u g h l y2 . S u p p o s e a g i v e n t e s t w i l l i d e n t i f y 9 5 o f p e o p l e

i The term sampling error s used here to represent anydi f ference between s ta t is t ics based on samples drawn f romthe same population (cf. Hu nter & Schmidt, 1990).2 W hile no o ne argues with the prev ious sentence, th e

present sentence m ay rankle for some. I t seems to us , how-ever , tha t a d iscuss ion of the probabi l i ty of depar ture f romnull implies that the null is taken to be true, and such adiscussion with such an implication is equivalent to a dis-cuss ion of the probabi l i ty of a result g iven the t ru th of thenull.


5/12

LOGIC AND THE NULL 165with schizophrenia as being schizophrenic and willidentify 97% of "normal" individuals (in the clinicalsense) as normals. The data or empirical outcomes, inthis case, are the results of the test for schizophrenia.The hypotheses correspond to the true nature of thetestee. So, P(normal test resultlnormal testee) = .97,while P (schizophrenic test resultlschizophrenictestee) = .95. Stated formally, P DolHo) = .97 andP DIIHI) = .95, while P DIlHo) = .03 and P DolH1)= .05. We also have the prior probability or base rateof occurrence for the null hypothesis, P(Ho) = .98.What we want to know, of course, is the probabilitythat a given person truly is schizophrenic in light ofthe test result. Formally, we want P HolD1) orP HIIDo). The Bayesian equation for P HoID1) s 3

P / - /o ) * e D l l / - / o )P ( H o l D 1 ) = P ( H o ) P ( D I l H o ) + P H 1 ) P(DolH~)

Using this equation, Cohen (1994) showed thatP HolD1) = .607. Substantively, if the test says that aperson is schizophrenic, then the person will actuallyturn out to be schizophrenic only 39% of the time. Ofthe positive test results, 61% are wrong

The point that we wish to make, however, is thatthis result is problematic from only one perspective,namely, that of the person who wants to find schizo-phrenics, Consider another perspective: that of a per-son in charge of hiring police officers. Such a personwould, in most cases, try to avoid hiring people whoseschizophrenia (or any other attribute) would be de-bilitating with respect to job performance. In otherwords, the employer's purpose is to make sure that agiven applicant is, in fact, normal in the clinical senseof the word. The test from Cohen's (1994) exampleidentifies 95 of every 100 people as normal, of whom94.9 really are normal. Therefore, the conditional ac-curacy of a normal result f rom the test (i.e., P HolDo))is 94.9/95, or .999 Only 1 out of every 1,000 peoplewith a normal result would actually suffer fromschizophrenia.

So, is this test useful to an employer? Without thetest, the employer would hire 20 people with schizo-phrenia in every 1,000 hiring decisions. With the test,the employer would get 1 person with schizophreniain every 1,000 hiring decisions. Thus, the odds ofhiring a person with schizophrenia without the test aregreater by a factor of 20 The point is that, not sur-prisingly, the context largely determines the problemscaused by interpreting the results of NHST in a certainway. Of course, we are only echoing statements made

almost 40 years ago. Rozeboom (1960) pointed outthat the probabilities associated with our hypothesesare not the only considerations when deciding wheth-er or not to accept or reject hypotheses. We must alsoconsider the "utilities of the various decision out-comes" (Rozeboom, 1960, p. 423). For certain typesof decisions, a procedure that is prone to mistakes ofone kind can be devastating because of the utilitiesassociated with those mistakes, whereas a differentprocedure that is prone to different kinds of mistakescan be quite useful because the mistakes that it makesresult in "missing on the safe side."Syllogistic Reasoning andProbabilistic Statements

It has been suggested that the logic of NHST is, ifyou will, illogical. Consider the issue as presented byCohen (1994), who set up various syllogisms repre-senting different ways of viewing the logic of hypoth-esis testing. Cohen (1994) pointed out that while therule of Modus Tollens can be universally applied topremises of the form "If A then B, not B," resultingin the conclusion "Not A," it cannot be universallyapplied to the premises, "If A then probably B, notB" to conclude "Probably not A." This sequence ofstatements is meant as an analog for the statementsthat are implicit in NHST. If the null were true, thena sample taken from the population associated withthe null would probably produce a statistic within acertain range (i.e., If A, then probably B). The statisticfrom our sample is not within that range (i.e., not B).Ergo, a population associated with the null valueprobably did not produce our sample (i.e., probablynot A). Cohen then gave an intriguing example thathighlights one of the problems that can arise as aresult of applying the Modus Tollens to probabilisticstatements. The example is as follows:

If a person is an American, then that person is probablynot a member of Congress.This person is a member of Congress, therefore,This person is probably not an American.

In this case, the two premises are perfectly true, andyet Modus Tollens fails to lead us to a reasonableconclusion.

As we stated above, this example is intriguing, but

3 Conditional probabilities of other hypotheses can becomputed using similar equations.


6/12

1 6 6 C O R T 1 N A A N D D U N L A Pi t i s l i m i t e d i n i t s ge ne r a l i z a b i l i t y f o r t wo r e a s ons . I t i si m p o r t a n t t h a t t h e s e r e a s o n s b e u n d e r s t o o d s o t h a t t h es i t u a ti o n s i n w h i c h t h e l o g i c o f N H S T i s a n d is n o tq u e s t i o n a b l e c a n b e i d e n t i f i e d . F i r s t , t h e c o n s e q u e n to f t h e f i r s t p r e m i s e , T h a t p e r s o n i s p r o b a b l y n o t am e m b e r o f C o n g r e s s , i s t r u e i n an d o f i ts e lf . A n yg i v e n p e r s o n i s p r o b a b l y n o t a m e m b e r o f C o n g r e s s .A s a r e s u l t , o n e c o u l d u s e a l m o s t a n y t h i n g a s t h ea n t e c e d e n t o f th i s p r e m i s e w i t h o u t d a m a g i n g t h e a c -c u r a c y o f t h e p r e m i s e. I f o n e o u t o f e v e r y t hr e e c o w si s b l u e , t h e n t h i s p e r s o n i s p r o b a b l y n o t a m e m b e r o fC o n g r e s s i s j u s t as v a l i d as t h e f i r s t p r e m i s e f r o mC o h e n ' s ( 1 9 9 4 ) e x a m p l e . W e e x p l a i n t h e i m p o r t a n c eo f t h is f e a t u r e o f t h e e x a m p l e m o m e n t a r i l y .

T h e s e c o n d l i m i t i n g a s p e c t o f th e f i r st p r e m i s e i st ha t wh i l e t he f i r s t p r e m i s e i s t r ue a s i t s t a nds , i t i s a l s ot h e c a s e t h a t b e i n g a n A m e r i c a n i s a n e c e s s a r y c o n -d i ti o n o f b e i n g a m e m b e r o f C o n g r e s s . I n o t h e r w o r d s ,w h i l e i t i s t r u e th a t I f a p e r s o n is a n A m e r i c a n , t h e nt h a t p e r s o n is p r o b a b l y n o t a m e m b e r o f C o n g r e s s , i ti s a l s o t r u e if a p e r s o n i s a m e m b e r o f C o n g r e s s , t h e nt h a t p e r s o n h a s t o b e a n A m e r i c a n . I t is b e c a u s e o ft h e s e t w o a s p e c t s o f t h e p a r t ic u l a r e x a m p l e c h o s e nt h a t t h e M o d u s T o l l e n s b r e a k s d o w n . C o n s i d e r a d i f -f e r e n t e x a m p l e , o n e t h a t is m o r e r e p r e s e n t a t i v e o f p s y -c h o l o g y :

I f Sample A were f rom some speci f ied popula t ion ofno r m a l s , t he n Sa mpl e A p r oba b ly wou l d no t be 50%schizophrenic.Sample A comprises 50% schizophrenic individuals ;therefore,Sample A is probably not f rom the no rm al popula t ion

I n t h is e x a m p l e , t h e c o n s e q u e n t o f th e f i rs t p r e m i s es t il l s t a nds by i t s e l f , t ha t is , a g i ve n s a m p l e o f pe o p l ep r o b a b l y w i l l n o t c o m p r i s e 5 0 % p e o p l e w i t h s c h i z o -p h r e n i a . H o w e v e r , t h i s s t a t e m e n t i s p a r t ic u l a r l y t r u e i ft h e a n t e c e d e n t h o l d s ; w h e r e a s , i n t h e C o h e n e x a m p l e ,t h e c o n s e q u e n t i s p a r t i c u l a r l y t r u e i f t h e a n t e c e d e n td o e s n o t h o l d . T o c l a r i f y t h i s , c o n s i d e r a t h i r d e x -a m p l e :

f f the p lanets revolve around the sun, then Sample Aprobably w ould not be 50% schizophrenic .Sample A comprises 50% schizophrenic individuals ;thereforeThe planets probably do not revolve around the sun.A s w i t h C o h e n ' s e x a m p l e , t h e c o n c l u s i o n h e r e i sf a l se . M o d u s T o l l e n s f a i l s t o l e a d t o a r e a s o n a b l e c o n -c l u s i o n b e c a u s e t h e t r u t h o f t h e a n t e c e d e n t o f t h e f ir s tp r e m i s e i s u n r e l a t e d t o t h e t r u t h o f it s c o n s e q u e n t . S o ,

w h i l e i t i s t h e c a s e t h a t M o d u s T o U e n s c a n n o t b ea p p l i e d t o p r o b a b i l i s ti c p r e m i s e s w h e n t h e t r u t h o f t h ea n t e c e d e n t o f t h e fi r s t p r e m i s e i s u n r e l a t e d o r n e g a -t i v e l y r e l a te d t o t h e t r u t h o f th e c o n s e q u e n t o f t h ep r e m i s e , i t i s a p p r o x i m a t e l y c o r r e c t f o r a n d c a n b ea p p l ie d t o a r g u m e n t s , t y p i c al o f p s y c h o l o g y , i n w h i c ht h e t r u th o f t h e tw o c o m p o n e n t s o f t h e f i rs t p r e m i s ea r e p o s i t i v e l y r e l a t e d . I n o t h e r w o r d s , t h e t y p i c a l a p -p r o a c h t o h y p o t h e s i s t e s t i n g d o e s n o t v i o l a t e t h e r e l -e v a n t r u l e o f s y l l o g i s t i c r e a s o n i n g t o a n y g r e a t d e g r e e .C o h e n ' s ( 1 9 9 4 ) e x a m p l e w a s u s e f u l i n th a t it s h o w e dw h y a p p l i c a t io n o f M o d u s T o l l e n s t o p r o b a b i li s ti cs t a t e m e n t s c a n b e p r o b l e m a t i c , b u t i t s h o u l d n o t b et a k e n t o m e a n t h a t t h is r u l e o f s y l l o g i s t i c r e a s o n i n g i su s e le s s f o r p s y c h o l o g y .

Interpretation of rror RatesT h i s b r i n g s u s t o t h e i s s u e o f in t e r p r e t a ti o n o f T y p e

I a n d T y p e I I e r r o r r a te s , u s u a l l y r e p r e s e n t e d a s e t a n d13, r e s p e c t i v e l y . T h e r e s e e m s t o b e s o m e c o n f u s i o n a st o th e m e a n i n g o f th e s e v a l u es . F o r e x a m p l e , S c h m i d t( 1996 ) s t a t e d r e pe a t e d l y t ha t t he Type I e r r o r r a t e , ~x ,i s z e r o , a s o p p o s e d t o . 0 5 o r . 0 1 , o r w h a t e v e r t h ep r e d e t e rm i n e d c u t o f f v a l u e i s. C o h e n ( 1 9 9 4 ) m a d es i m i l a r s t a te m e n t s . T h e i r r e a s o n i n g i s t h a t b e c a u s e t h eh y p o t h e s i s o f n o e f f e c t i s n e v e r p r e c i s e l y t r u e , i t i s n o tp o s s i b l e t o f a l s e l y r e j e c t t h e n u l l h y p o t h e s i s ( s e eF r i c k , 1 9 9 5 , f o r a n a l t e r n a t i v e p o s i t i o n ) . I n o t h e rw o r d s , t h e n u l l i s a l w a y s f a l s e , s o r e j e c t i n g t h e n u l lc a n n o t b e a n e r r o r . T h i s m a y b e t r u e , b u t i t h a s n o t h -i n g t o d o w i t h t h e T y p e I e r r o r r a t e .

The Type I e r r o r r a t e , o r , i s t he p r oba b i l i t y t ha t t hen u l l w o u l d b e r e j e c t e d i f t h e n u l l w e r e t r u e . N o t e t h a tt h e r e i s n o s u g g e s t i o n h e r e t h a t t h e n u l l i s o r i s n o tt r u e . T h e s u b j u n c t i v e were i s u s e d i n s t e a d o f is tod e n o t e t h e c o n d i t i o n a l n a t u r e o f t h is p r o b a b i l i t y . T h eT y p e I e r r o r r a t e i s t h e p r o b a b i l i t y t h a t t h e h y p o t h e t i -c a l n u l l d i s t r i b u t i o n w o u l d p r o d u c e a n o b s e r v e d v a l u ewi t h a c e r t a i n e x t r e m e ne s s . I f t h i s va l ue i s s e t a t . 05 ,t h e n i n o r d e r f o r t h e o b s e r v e d t e s t r e s u l t t o b e c o n -s i de r e d s t a t i s t i c a l l y s i gn i f i c a n t , i t wou l d ha ve t o be av a l u e s o e x t r e m e t h a t i t ( o r a v a l u e m o r e e x t r e m e )w o u l d o c c u r 5 % o f t h e t im e o r l e ss i f w e r e p e a t e d l ys a m p l e d f r o m a n u l l d i s tr i b u t io n . T h e . 05 v a l u e i s t h eT y p e I e r r o r r at e , r e g a r d l e s s o f w h e t h e r o r n o t t h e n u l li s t r ue . E v e n i f w e k n o w t h e n u l l t o b e f a l s e , th e T y p eI e r r o r r a t e i s s t i l l . 05 be c a us e i t ha s t o do w i t h ah y p o t h e t i c a l d i s t r i b u t i o n , n o t t h e a c t u a l s a m p l i n g d i s -t r i bu t i on o f t he t e s t s ta t i s ti c . A l ph a i s no t t he p r ob -a b i l i ty o f m a k i n g a T y p e I e rr o r . I t i s w h a t t h e p r o b -


7/12

LOGIC AND THE NULL 167ability of making a Type I error would be if the nullwere true. One can, perhaps, argue that the term T y p eI e r ror ra t e is misleading. A better term might becond i t i ona l T ype I e r ror ra t e . Regardless of the termused, however, the value that we choose for ct is theType I error rate regardless of the truth of the null

This is not to say that the Type I error rate is theonly error rate on which we should focus or that thiserror rate alone allows one to determine the impor-tance of our empirical results. The Type II error rate,which is also a conditional error rate, is at least asimportant as the Type I error rate. Furthermore, as iswell known, for large sample sizes, the null can berejected regardless (almost) of the effect size and re-gardless of the Type I error rate that is chosen. Thus,the Type I error rate is only one of many consider-ations in a test of significance. Nevertheless, it is im-portant that the meaning of these values and theirconditional nature be understood so that further mis-interpretation does not occur.

Another issue with respect to interpretation of errorrates has to do with the nul l versus nil hypo thesisdistinction. As pointed out by Cohen (1994), the termnul l hypo t hes i s receives its name by virtue of the factthat it is the hypothesi s to be nullified Thus, the valueassociated with this hypothesis need not be zero. Itcan be any value against which we wish to comparethe empirical result. The ni l hypo t hes i s according toCohen (1994), would be a null hypothesis for whichthe value to be nullified is precisely zero. The pointthat Cohen (1994), Thompson (1992), and others havetried to make is that because the nil hypothesis isalways false, there is no glory in rejecting it. It is a

stra w ma n that is set up for the purpose of beingknocked down. Their point is well taken (althoughFrick, 1995, and others have argued that there aresituations in which the nil can be precisely true), butsome authors have taken it further than it can go. Forexample, Schmidt (1996) used the position that the nilis always false to suggest that all research that hascompared a research hypothesis with the nil hypoth-esis is worthless (except as fodder for meta-analyses).While it is certainly true that the social sciencesshould expand their methodological thinking to in-clude null hypotheses other than the nil hypothesis, itis not true that the use of the nil renders previousresearch worthless. For example, suppose that thereare theoretical reasons for positing a positive relation-ship between two variables. In an attempt to investi-gate this relationship, data from 100 subjects are col-lected, and the correlation is found to be .40 This

value can be converted into a t score (4.32), which isgreater than any cutoff value that is likely to be rel-evant for our choices of test and significance level.Thus, it is highly unlikely that this result would haveoccurred if the nil hypothesis were true. We would,therefore, proceed as if the nil were false and theresearch hypothesis were true pending further infor-mation (cf. Neyman & Pearson, 1928, 1933)

However, as pointed out by others, the nil is some-thing of a straw man. It allows one to address thequestion, Ho w likely is it that a population with acorrelation of zero would produce a given sample-based correlati on? It would be more interesting toask whether or not it is likely that a population with atrivial correlation would produce a given sample-based correlation. While it is true that one person'swhopping effect is another person's trivium, let usassume for the moment that any variable that explainsless than 1% of the variance in another variable ex-plains only a trivial amount of variance. Instead ofusing the nil hypothesis, we might use a null value of.10 (which is the square root of .01). Thus, we wouldcompare our observed value of .40 with the null value10 instead of the nil value This test requires that weconvert both our observed correlation and our nullvalue to z scores with the Fisher r to z transformation,which yields z values of 04236 and 0.1003. We thencompute a z value representing the difference betweenthese values. For the present example, this z value is3.185, which is also greater than any cutoff value thatis likely to be relevant for our choices of test andsignificance level. Thus, the outcome is the same forthis test as it was for the test involving the nil hypoth-esis: We would proceed as if the research hypothesiswere true pending further information. This is not tosay that it makes no difference which value we chooseas the null value Instead, the point that we wish tomake is that it is nonsensical to suggest that the use ofzero as the null value has produced nothing but worth-less research. The point nil can be thought of as themidpoint of some interval that (a) includes all valuesthat might be considered trivial and (b) is smallenough that calculations using the point nil give agood approximation of calculations based on othervalues within the interval. This interval is analogousto the good- enou gh belt described by Serlin andLapsley (1985). Of course, the explicit use of such abelt would be preferable to simply assuming that itprovides support for a given hypothesis. Nevertheless,our point is that the conclusions drawn from the vastmajority of research that has focused on the nil would


8/12

1 6 8 C O R T I N A A N D D U N L A Ph a v e b e e n v e r y m u c h t h e s a m e e v e n i f a n a l te r n a ti v en u l l v a l u e h a d b e e n u s e d .

A d d i t i o n a l l y , w h i l e i t m a y b e t h e c a s e t h a t t h e n i lh y p o t h e s i s i s a l w a y s f a l s e a n d t h a t T y p e I e r r o r s w i t hr e s p e c t t o n i l h y p o t h e s e s n e v e r o c c u r , t h e s a m e c a n b es a i d o f a n y h y p o t h e s i s r e l a t i n g t o a s p e c i f i c p o i n t i n ac o n t i n u u m ( F r i c k , 1 9 9 5 ) . T h i s f a c t d o e s n o t a ll o w o n et o c o n c l u d e t h a t a p o i n t n u l l h y p o t h e s i s i s a s t r a wm a n . R e j e c t i o n o f a g i v e n n u l l h y p o t h e s i s i m p l i e s t h er e j e c t i o n n o t o n l y o f th e p a r t i c u l a r n u l l v a l u e i n q u e s -t i o n , b u t a l s o o f a l l o f t h e v a l u e s i n t h e e n d o f t h ed i s t r i b u t i o n t h a t i s o p p o s i t e t o t h e e n d i n w h i c h t h eo b s e r v e d v a l u e r e s i d e s . F o r e x a m p l e , i f a n o b s e r v e dc o r r e l a t i o n o f . 4 0 i s c o m p a r e d w i t h t h e n u l l v a l u e o fz e r o , a n d N H S T l e a ds t o t h e r e j ec t i o n o f t he h y p o t h -e s i s p = 0 , t he n i t a l s o l e a ds t o the r e j e c t i on o f t heh y p o t h e s e s ( p = - . 0 1 , p = - . 0 2 , p = - . 1 0 , a n d s of o r t h o n t o p = - 1 . I f t h e n u l l w e r e i n s t e a d p = . 1 0 ,t h e n r e j e c t i o n o f t h is h y p o t h e s i s w o u l d a l s o e n t a ilr e j e c t i o n o f h y p o t h e s e s , r e l a t i n g t o . 0 9 , . 0 8, a n d s of o r t h , o n t o - 1 . S i m i l a r r e a s o n i n g c a n b e a p p l i e d t od i f f e r e n c e s b e t w e e n m e a n s , o r w h a t e v e r t h e p a r a m -e t e r o f i n te r e s t. O u r p o i n t i s t h a t w h i l e t h e p v a l u es p e c i f i c a l l y a p p l i e s t o a h y p o t h e t i c a l d i s t r i b u t i o nb a s e d o n t h e n u l l v a l u e o n l y , a n d w h i l e t h i s d i s t r i b u -t i o n m a y n e v e r , i n f a c t , e x i s t, i t c a n n o t b e c l a i m e d t h a td i s t r i bu t i ons r e l a t i ng t o a l l va l ue s i n t he oppos i t e d i -r e c t i o n f r o m t h e o b s e r v e d r e s u l t d o n o t e x i s t . S i n c et h e s e m o r e e x t r e m e d i s t r i b u t i o n s w o u l d y i e l d e v e ns m a l l e r p v a l u e s , w e a r e e v e n m o r e j u s t i f ie d i n r e j e c t -i n g h y p o t h e s e s r e l a t i n g t o t h e n u l l v a l u e s a s s o c i a t e dw i t h t h e s e d i s t r i b u t i o n s t h a n w e a r e i n r e j e c t i n g h y -p o t h e s e s r e l a t in g t o t h e n u l l v a l u e o f d i r e c t i n t e re s t .

T h i s a r g u m e n t i s m o r e e a s i l y u n d e r s t o o d i n t h e c o n -t e x t o f d i r e c t i o n a l h y p o t h e s i s t e st s . 4 I f a t h e o r y s u g -g e s t s a n e g a t i v e c o r r e l a t i o n b e t w e e n t w o v a r i a b l e s ,t h e n o n e m i g h t u s e a s i g n i f i c a n c e t e s t i n w h i c h o n l yt h e n e g a t i v e e n d o f t h e d i s t r ib u t i o n i s t a r g e t e d . I f t h er e l e va n t s t a t i s t i c f a l l s w i t h i n t he r e j e c t i on r e g i on , t he nt h e h y p o t h e s i s a s s o c i a t e d w i t h t h e n u l l v a l u e i s r e -j e c t e d ( t e n t a t i ve l y ) . B y i mp l i c a t i o n , a l l o f t he d i s t r i-b u t i o n s a s s o c i a t e d w i t h p o s i t i v e v a l u e s a r e a l s o r e -j e c t e d . I n d e e d , a s M e e h l ( 1 9 6 7 ) p o i n t e d o u t , t h e r e i sn o r e a s o n t o s u g g e s t t h a t a l l h y p o t h e s e s a s s o c i a t e dw i t h a g i v e n h a l f o f a d i s t r i b u t i o n a r e a l w a y s f a l s e .T h u s , t h i s s i g n i f i c a n c e t e s t i s n o t t r i v i a l . T h e t h e o r yp l a c e s c e r t a i n c o n s t r a i n t s o n t h e p a r a m e t e r o f i n te r e s t,t h a t i s , p o s i t i v e o r n e g a t i v e , a n d t h e s i g n i f i c a n c e t e s ta l l o w s t h e r u l i n g o u t o f t h e h y p o t h e s e s a s s o c i a t e dw i t h t h e o p p o s i t e d i r e c t i o n .

T h e i s su e i s m o r e c o m p l i c a t e d f o r n o n d i r e c t i o n al

t e s t s . B e c a u s e s u c h t e s t s c o n t a i n s o m e r e j e c t i o n r e -g i o n i n b o t h t a i l s o f t h e d i s t r i b u t io n , i t h a s b e e n a r -g u e d t h a t a n e m p i r i c a l r e s u l t i n o n e e n d o f th e d i s t r i-b u t i o n d o e s n o t i m p l y r e j e c t i o n o f a l l v a l u e s i n t h eo p p o s i t e e n d o f t h e d i s tr i b u t io n . T o s h e d s o m e l i g h to n t h i s t o p i c , l e t u s f i r s t o b s e r v e t h a t n o n d i r e c t i o n a lt e s ts o c c u r i n t w o c o n t e x t s . I n t h e f i r s t, t h e t h e o r y i s s ow e a k t h a t i t c a n n o t s u g g e s t a d i r e c t i o n o r p a t t e r n .T h u s , th e r e is n o r e a s o n t o p r e f e r o n e d i r e c t i o no v e r th e o t h e r , a n d a n o n d i r e c t i o n a l h y p o t h e s i s i st e s t e d . I n t h e s e c o n d c o n t e x t , t h e t h e o r y b e i n g t e s t e dd o e s s u g g e s t a p a r t i c u l a r p a t t e r n o f r e l a t i o n s h i p s ,m e a n d i f f e r e n c e s , a n d s o o n , b u t t h e a n a l y s i s t e c h -n i q u e d o e s n o t a l l o w f o r p r e c i s e c o n s i d e r a t i o n o f t h e s ep a t t e r n s . I t i s t h i s c o n t e x t t o w h i c h o u r a r g u m e n t ss p e a k . F o r e x a m p l e , t h e n u l l h y p o t h e s i s i n a o n e w a ya n a l y si s o f v a ri a n c e ( A N O V A ) h a s t o d o w i t h e q u a li t yo f g r o u p m e a n s . T h i s n u l l c a n b e r e j e c t e d i f a n y f o r mo f d e p ar t u re f r o m e q u a l i t y o c c u rs . T h u s , a n A N O V Ar e p r e s e n t s a n o n d i r e c t i o n a l t e st , e v e n i f t h e t h e o r y i nque s t i on s ugge s t s a pa r t i c u l a r pa t t e r n . I t i s f o r t h i sr e a s o n t h a t t e s t s s u b s e q u e n t t o t h e A N O V A a r e p e r -f o r m e d . T h e N H S T a s s o c ia t e d w i t h th e o m n i b u sA N O V A a d d r e ss e s o n l y t h e is s u e o f e q u a l it y o fm e a n s . T h u s , i t r e p r e s e n t s a p r e l i m i n a r y s t e p i n t h ep r o c e s s o f a ss e s s i n g t h e d e g r e e t o w h i c h d a t a a n dt h e o r y a r e c o n s i s t e n t w i t h o n e a n o t h e r . T h e t e s t s s u b -s e q u e n t t o t h e A N O V A , w h i c h o f t e n a l s o i n c l u d eN H S T s , a d d r e ss t h e m o r e s p e c i fi c q u e s ti o n s c o n c e r n -i n g t h e p a t t e r n o f t h e re s u l ts . W i t h i n t h e c o n t e x t o ft h e s e m o r e s p e c i f i c t e st s , r e j e c t io n o f t h e n u l l i m p l i e sr e j e c t i on o f a l l d i s t r i bu t i ons t ha t a r e a t l e a s t a s un -l i ke l y a s t he nu l l d i s t r i bu t i on .

F i n a l l y , i t s h o u l d b e n o t e d t h a t e v e n i f T y p e I e rr o r sf o r p o i n t n u ll h y p o t h e s e s w e r e t e c h n i c a l l y i m p o s s i b l e ,i t i s e n t ir e l y p o s s i b l e t o c o m m i t e r r o r s t h a t a r e s i m i l a r ,i f n o t i d e n t i c a l t o , T y p e I e r r o r s b y c o n c l u d i n g t h a tt r i v i a l d e p a r t u r e s f r o m t h e n u l l v a l u e j u s t i f y t h e c o n -c l u s i o n t h a t t h e n u l l h y p o t h e s i s s h o u l d b e s p u m e d a n dt h e a l t e r n a t i v e h y p o t h e s i s a d o p t e d . F o r e x a m p l e , c o n -s i d e r t h e r e la t i o n s h i p b e t w e e n e x t r o v e r s i o n a n d j o bp e r f o r m a n c e f o r s a l e s r e p r e s e n t a t i v e s a s r e p o r t e d b yB a r r i c k a n d M o u n t ( 1 9 9 1 ) . T h e m e t a - a n a l y s i s - b a s e dc o r r e l a t i o n b e t w e e n t h e s e t w o v a r i a b l e s u n c o r r e c t e d

4 Our purpose i s no t to endorse o r recom men d againstdirectional hyp oth esis tests (see Harris, 1 994, for a discus-sion of the problems associated with one-tailed tests) . Wemention one-tailed tests only to clarify our point with re-spect to re jec t ion of se ts o f hypotheses .


9/12

L O G I C A N D T H E N U L L 1 6 9f o r a r ti f a c t s w a s . 0 9 . G i v e n t h e s a m p l e s i z e (2 , 3 1 6 ) ,s t a ti s t ic a l s i g n i f i c a n c e , n a m e l y , H o : p = 0 , w a s n o t a ni s s u e . I n o t h e r w o r d s , t h e n i l h y p o t h e s i s i s r e je c t e di p s o f a c t o . H o w e v e r , a d e c i s i o n t h a t i s , f o r p r a c t i c a lp u r p o s e s , a T y p e I e r r o r i s s ti ll p o s s i b l e . I f o n e c o n -c l u d e s f r o m t h i s e m p i r i c a l r e s u l t t h a t e x t r o v e r s i o n e x -p l a in s a m e a n i n g f u l a m o u n t o f v a ri a n c e i n j o b p e r f o r -m a n c e i n s p i t e o f th e f a c t t h a t l e s s t h a n 1 a p p e a r s t ob e e x p l a i n e d , a n d i f a c o r r e l a t i o n o f . 1 0 o r l e s s i sc o n s i d e r e d t r i v i a l, t h e n t h i s i n t e r p r e ta t i o n c o u l d e a s i l yb e c o n s t r u e d a s a n e r r o r o f s o m e s o r t . I n f a c t , t h ea u t h o r s d i d c o n c l u d e t h a t e x t r o v e r s i o n w a s a v a l i dp r e d i c t o r o f j o b p e r f o r m a n c e f o r s a l e s r e p re s e n t a t i v e s .T h u s , e v e n i n t h o s e c a s e s i n w h i c h a T y p e I e r r o r i nt h e s t r i c t s e n s e i s, f o r a l l i n t e n t s a n d p u r p o s e s , i m p o s -s i b l e , i t i s p o s s i b l e t o e v a l u a t e c o n c l u s i o n s w i t h r e -s p e c t t o T y p e I e r r o r s , v i s ~ v i s a l t e r n a t i v e n u l l h y -p o t h e s e s .

W h a t i f S ig n i f ic a n c e T e s t s A r e A b a n d o n e d ?V a r i o u s a u t h o r s ( e . g . , C o h e n , 1 9 9 4 ; S c h m i d t , 1 9 9 6 )

h a v e a r g u e d t h a t N H S T s h o u l d b e o u t l a w e d a n d r e -p l a c e d b y p a r a m e t e r e s t i m a t i o n p r o c e d u r e s s u c h a sc o n f i d e n c e i n t e r v a l s ( C I s ) . ~ T h i s a r g u m e n t , h o w e v e r ,b e c o m e s m o o t w i t h a c l e a r u n d e r s t an d i n g o f a lp h a .S u p p o s e , f o r e x a m p l e , t h a t w e w i s h t o c o m p a r e as a m p l e , w i t h a m e a n o f 1 , 1 8 0 a n d a s t a n d a r d e r r o r o ft h e m e a n o f 8 .1 7 , t o a p o p u l a t i o n w i t h a m e a n o f1 , 11 0 . I n a t e s t o f s i g n i f i c a n c e , a l p h a i s t h e p r o b a b i l i t yt h a t t h e p o p u l a t i o n w i t h t h e m e a n o f t he 1 , 11 0 w o u l dh a v e p r o d u c e d a s a m p l e w i t h a m e a n o f s u c h a s iz et h a t w e w o u l d r e j e c t t h e n u l l h y p o t h e s i s . I f o n e d i s -c a r d s t e s t s o f s i g n i f i c a n c e , a l p h a i s e f f e c t i v e l y e q u a lt o z e r o b e c a u s e n o n u l l h y p o t h e s i s w i l l e v e r b e r e -j e c t e d .

T h e w i d t h o f a C I is d e t e r m i n e d b y 1 - e t ( m u l t i -p l i e d b y 1 0 0 t o e x p r e s s i t a s a p e r c e n t a g e ) . F o r t h ee x a m p l e a b o v e , t h i s a l p h a v a l u e w o u l d b e t h e p r o b -a b i l i t y t h a t a p o p u l a t i o n w i t h a m e a n o f 1 , 1 8 0 w o u l dp r o d u c e a s a m p l e w i t h a m e a n o f a c er t a in m a g n i t u d e( + t~ * S E ) . I n b o t h c a s e s , a l p h a p a r t i a l l y d e t e r m i n e sw h i c h v a l u e s w i l l f a l l i n t o o n e c l a s s v e r s u s a n o t h e r .F o r t h e s i g n i f i c a n c e t e s t , a l p h a p a r t i a l l y d e t e r m i n e sw h i c h v a l u e s a r e a s s o c i a t e d w i t h a d e c i s i o n t o c o n -c l u d e t h a t th e m e a n o f th e p o p u l a t i o n f r o m w h i c h t h es a m p l e c a m e i s d i f f e r e n t f r o m 1 , 11 0 . F o r t h e C I , a l p h ap a r t i a l l y d e t e r m i n e s w h i c h v a l u e s r e p r e s e n t m e a n s o fs a m p l e s t h a t a r e l i k e ly , o r a t l e a s t n o t u n l i k e l y , t o h a v eb e e n p r o d u c e d b y a p o p u l a t io n w i t h a m e a n o f 1 , 1 80 .I n u s i n g a c o n f i d e n c e i n t e r v a l, w e r e j e c t, i m p l i c i t l y o r

e x p l i c i t l y , a n y h y p o t h e s e s a s s o c i a t e d w i t h v a l u e s t h a tl i e ou t s i de t he i n t e r va l . I t s hou l d be no t e d t ha t t h i s i st r u e r e g a r d l e s s o f t h e w a y t h a t t h e i n t e rv a l i s d e -s c r i b e d . E v e n i f o n e ' s f o c u s i s o n t h e s a m p l i n g e r r o r -b a s e d b a n d a b o u t a p o i n t p a r a m e t e r e s t i m a t e ( a s o p -p o s e d t o f o c u s i n g o n t h e v a l u e s t h a t d o n o t l i e w i t h i nt h e b a n d ) , t h e b o u n d a r i e s o f t h is b a n d a r e d e f i n e d b yt h e v a l u e s t h a t l i e o u t s i d e i t. T h e p r e s e n t a t i o n o f s u c ha b a n d n e c e s s a r i l y i m p l i e s t h e e x c l u s i o n o f c e r t a inv a l u e s . S o m e t i m e s , t h i s e x c l u s i o n w i l l b e i n e r r o r , a n dt h e p r o b a b i l i t y o f s u c h a n e r r o r i s c a l l e d a l p h a .

C o n s i d e r w h a t h a p p e n s i f a l p h a i s s e t t o z e r o f o rc o n f i d e n c e i n t e r v a ls , a s i t w o u l d b e f o r N H S T i fN H S T w e r e a b o l is h e d . A n a l p h a v a l u e o f z e r o s u g -g e s t s a 1 0 0 C I , w h i c h w o u l d r a n g e f r o m m i n u s i n -f i n i t y t o p l u s i n f i n i t y ( f r o m - 1 t o 1 f o r c o r r e l a t i o n s ) .S u c h C I s , a l th o u g h p e r f e c t i n t h e s e n s e o f a l w a y sb e i n g c o r r e c t , a r e o f n o u s e . T o m a k e t h e C I u s e f u l i na n y s e n s e , w e m u s t a c c e p t a n i m p e r f e c t C I .

H i s t o r i c a l l y , w i t h t h e c o n c e p t o f p r o b a b l e e r r o r ,e a r l y s t a ti s t ic i a n s u s e d w h a t w e r e b a s i c a l l y 5 0 C I s .T h a t p r a c t ic e w a s a b a n d o n e d i n fa v o r o f m o r e c o n -s e r v a t i v e C I s f o r w h i c h f a i l u r e o f t h e i n t e r v a l t o c o n -t a i n t h e p o p u l a t i o n p a r a m e t e r w a s c o n s i d e r a b l y l e s sl i k e l y t h a n 5 0 . F u n d a m e n t a l l y , t h e r e is n o w a y t oa v o i d m a k i n g a d e c i s i o n r e g a r d i n g a l p h a , w h e t h e r t h a td e c i s i o n i n v o l v e s N H S T o r C I s . I f w e s e t a l p h a a t z e rof o r N H S T ( r e fu s e t o d o t h em ) , w e s h o u l d a l s o d o s of o r C I s , w h i c h s h o w s t h e u n d e r l y i n g f a l l a c y o f t h er e p l a c e m e n t o f N H S T w i t h C I s . A s w i t h s i g n i f ic a n c et e s ts , t h e r e is a p r o b a b i l i t y o f b e i n g w r o n g w h e n f o r m -i n g a C I , a n d t h a t p r o b a b i l i t y i s c a l l e d a l p h a .

L i k e w i s e , i f te s t s o f s i g n i f i c a n c e a r e a b o l i s h e d ,p o w e r , t h e p r o b a b i l i t y o f re j e c t i n g t h e n u l l h y p o t h e s i sw h e n i t i s f a l s e , i s z e r o . P o w e r , l i k e C I s , d e p e n d s o na l pha ; w i t ho u t a l pha , be t a is 100 a nd po w e r i s 1 - [3 .T h u s , p o w e r e q u a l s z e r o. W h e n t h e se p o i n t s a r e a d d e dt o t h e f a c t t h a t C I s a n d s i g n i f i c a n c e t e s ts a r e b a s e d o np r e c i s e l y t h e s a m e i n f o r m a t i o n ( i . e . , p a r a m e t e r e s t i -m a t e s a n d s t a n d a r d e r r o r v a l u e s ) , t h e o n l y r e a s o n a b l ec o n c l u s i o n i s t h at C I s a n d p o w e r e s t im a t i o n c a n n o t b ed o n e i n s t e a d o f t e s ts o f s i g n i f i c a n c e b u t t h a t i n s te a dt h e y s h o u l d b e d o n e i n c o n j u n c t i o n w i t h s i g n i fi c a n c etes ts .

The re are ma ny statist ics---such as goodn ess-of-fi t in-dices , test s of normal i ty , and tes ts o f ran dom nes s- - fo rwhich confidence intervals are not available (Nantrella,1972).


10/12

170 CORT I NA AND DUNL APD i s c u s s i o n

I n conc l us i on , we woul d l i ke t o r e i t e r a t e our agr ee -m e n t w i t h m a n y o f t h e p o s i ti o n s t a k e n b y t h e a u t h o r sl i s ted i n t he f i r s t pa r agr a ph o f t h i s a r ti c l e . S i gn i f i cancet e s ti n g i s a b u s e d . A p p l i c a ti o n o f M o d u s T o l l e n s a n do t he r r u l e s o f sy l l og i s t i c r eason i ng t o p r obab i l i s t i cs t a t ement s ca n l ead t o p r ob l e ms . I n t e r p r e t a t i on o f pva l ues a s t he p r obab i l i t y o f t he t r u t h o f t he nu l l hy-p o t h e s i s g i v e n t h e d a t a i s i n a p p r o p r i a t e . C I s a n dp o w e r s h o u l d b e r e p o r t ed . T h e p o i n t s t h at w e w i s h t oma ke he r e a r e t hese . F i r st , t he pur p ose o f da t a ana l ys i si s t o a l l ow us t o exami ne t he ex t en t t o whi ch t he da t ap r o v i d e c o r r o b o r a t i o n f o r t h e t h e o r y - b a s e d a n s w e r t ot h e r e s e a r c h q u e s t i o n . T h i s c o r r o b o r a t i o n t y p i c a l l yc o m e s i n t h e f o r m o f d i s c o n f i r m a t i o n o f a l t e r n a t i v eh y p o t h e s e s a n d e x p l a n a t i o n s ( P o p p e r , 1 95 9 ). N H S Tg i v e s u s a n o b j e c t i v e m e c h a n i s m b y w h i c h w e c a nr u l e ou t hypo t heses and exp l ana t i ons r e l a t i ng t o t hen u ll . S e c o n d , t h e a r g u m e n t s a g a i n s t th e u s e o f N H S Ta r e b u i lt o n f a u l t y p r e m i s e s , m i s l e a d i n g e x a m p l e s , a n dmi sunder s t and i ng o f ce r t a i n c r i t i ca l concep t s . W e a t -t e m p t t o s h o w t h a t t h e r e a r e m a n y c a s e s i n w h i c hd r a w i n g c o n c l u s i o n s a b o u t h y p o t h e s e s b a s e d o n p v a l -ues i s pe r f ec t l y r easonab l e . I ndeed , a p r obab i l i s t i cv e r s i o n o f t h e M o d u s T o l l e n s r u l e o f s y l l o g i s ti c r e a -s o n i n g c a n b e a p p l i e d t o m a n y e x a m p l e s t y p i c a l o fp s y c h o l o g y t o p r o d u c e a p p r o x i m a t e p r o b a b i l i s t i cs t a t em e n t s a b o u t h y p o t h e s e s . F u r t h e r m o r e , a f i r m u n -de r s t and i ng o f t he na t u r e o f e r r o r r a t e s g i ves i ns i gh ti n t o t he f ac t tha t p va l ues a r e use f u l r eg a r d l e s s o f t heac t ua l s t a t e o f r ea l i ty . F i na l l y , t he pos i t i on t ha t NH S Ts h o u l d b e r e p l a c e d b y C I s i s n o n s e n s ic a l . T h e t w o a r eb a s e d o n e x a c t l y t h e s a m e i n f o r m a t i o n , a n d b o t h i n -v o l v e a n e x c l u s i o n a r y d e c i s io n o f s o m e k i n d . T o c r it i-c i ze and r ev i l e one whi l e advoca t i ng t he o t he r i s ne i -t he r cons i s t en t nor r a t i ona l .

F u t u re o f D a t a A n a l y s i sT h i s i s n o t t o s a y t h a t N H S T i s a p p r o p r i a t e f o r

eve r y s i t ua t i on . I t i s no t . Bu t ne i t he r a r e t he a l t ema-f i ves appr opr i a t e f o r eve r y s i t ua t i on , and ne i t he r a r et h e y t o b e a p p l i ed w i t h o u t j u d g m e n t . V a r i o u s s u g g e s -t io n s h a v e b e e n m a d e w i t h r e s p e c t t o m e t h o d s t h a t a r ed e e m e d s u p e r i o r t o N H S T s u c h a s C I s , e f f e c t s i z ee s t im a t i o n s , a n d m e t a - a n a l y s is . E a c h o f t h e se m e t h o d sce r t a i n l y has i ts advan t ages , bu t t hey a r e no l e s s p r onet o a b u s e t h a n a n y o t h e r m e t h o d , a n d n o n e o f t h e m i sa p p r o p r i a t e f o r e v e r y s i t u a t i o n . F o r e x a m p l e , m e t a -ana l ys i s i s no t use f u l f o r man y r e sea r ch ques t i ons t ha th a v e n o t b e e n a d d r e s s e d e m p i r i c a l l y in p r e v i o u s s t u d -

i e s, a n d l i k e N H S T , i t h a s a s s u m p t i o n s t h a t s h o u l d b econs i de r ed when i n t e r p r e t i ng r e su l t s . E f f ec t s i ze e s t i -m a t i o n m u s t a l s o b e a p p r o a c h e d w i t h c a u t i o n , a s a ne f f e c t s iz e e s t i m a t e i s t y p i c a l ly t h e a m o u n t o f v a r i a n c ei n o n e v a r i a b l e a c c o u n t e d f o r b y a n o t h e r i n th e s a m p l ea t hand . T he p r ob l em i s t ha t e f f ec t s i ze e s t i ma t es a r edepen den t o n t he va r i ab i l it i e s o f t he pa r t i cu l a r mea-s u r e s a n d e x p e r i m e n t a l m a n i p u l a t i o n s u s e d i n t h es a m p l e ( C o r t i n a a n d D e S h o n , 1 9 9 7 ; D o o l i n g a n dDank s , 1975) , t he r e f o r e , t he use o f d i f f e r en t ma ni pu -l a t i ons o r measur es may r e su l t i n d i f f e r en t e f f ec t s i zees t i ma t es . E f f ec t s i ze e s t i ma t es a r e he l p f u l bu t mus tbe i n t e r p r e t ed w i t h cau t i on .

F i n a l ly , m a n y h a v e r e c o m m e n d e d C I s a s a r e p l a c e -m e n t f o r N H S T . H o w e v e r , c o n f i d e n c e i n t e r v a l s a n dN H S T c a l c u l a t i o n s a r e b a s e d o n p r e c i s e l y t h e s a m ei n f o r m a t io n . F o r e x a m p l e , a 9 5 C I a b o u t t h e d i f fe r -ence be t ween t wo means and a s i gn i f i cance t e s t a t . 05b o t h u s e t h e d i f f e r e n c e b e t w e e n s a m p l e m e a n s , t h eva l ue f r om t he t d i s t r i bu t i on cor r e spond i ng t o t he de -g r e e s o f f r e e d o m i n v o l v e d , th e s a m p l e v a r i a n c e s , a n dt h e s a m p l e s iz e s . T h e t w o m e t h o d s s i m p l y p r e s e n t t h isi n f o r m a t i o n i n d i f fe r e n t w a y s . T h e r e s u l t is a n e m p h a -s i s o n p a r a m e t e r e s t i m a t i o n v e r s u s a n e m p h a s i s o ns a m p l i n g e r r o r, w i t h e a c h e m p h a s i s h a v i n g i ts a d v a n -t a g e s a n d d i s a d v a n t a g e s.

A l so , t he r e i s no r eason t o be l i eve t ha t t he use o fCI s i ns t ead o f s i gn i f i cance t e s ts w i l l change any t h i ng .W e a r e a l l f ami l i a r w i t h s t ud i e s i n whi ch CI s a r er epor t ed , and t hese s t ud i e s o f t en po i n t ou t whe t he r o rno t t he i n t e r va l i nc l udes ze r o . T ha t i s a s i gn i f i cancet e s t . I t has been a r gued t ha t t h i s i s an i nappr opr i a t ea p p l i c a t i o n o f C I s a n d s h o u l d b e d o n e a w a y w i t h( S chm i d t , 1996) , thus l eav i ng on l y t he pa r am et e r e s -t i ma t e and t he w i d t h o f t he band a r ou nd i t. Unf or t u -na t e l y , no a l t e r na t i ves a r e o f f e r ed f o r ob j ec t i ve l y de -t e r m i n i n g t h e e x t e n t t o w h i c h t h e d a t a c o r r o b o r a t e t h et h e o r e ti c a l p r e d ic t i o n s o f t h e s t u d y . M o r e o v e r , m a n yo f t h o s e w h o w o u l d r e p l a c e s i g n i fi c a n c e t e s ts w i t h C I ss e e n o n e e d f o r a l t er n a t iv e s b e c a u s e t h e y b e l i e v e t h a tn o c o n c l u s i o n s c a n b e d r a w n f r o m t h e s i n g l e s a m p l es t u d y ( S c h m i d t , 1 9 9 6 ) . F o r t h o se o f u s w h o b e l i e v et ha t t he r e i s un i que mer i t i n t he s i ng l e s ampl e s t udy ,t h e l a c k o f c r i te r ia f o r d e t e r m i n i n g t h e a g r e e m e n t b e -t w e e n t h e o r y a n d d a t a i s p r o f o u n d l y d i s t u rb i n g . T h i sl a c k l e a v e s a y a w n i n g g a p i n t h e s y s t e m u s e d t o e v a l u -a t e th e m e r i t o f m o s t m a n u s c r i p t s s u b m i t t e d f o r p u b -l icat ion.

I n a n y c a s e , t h e s o r ts o f p r o b l e m s t h a t o c c u r w i t hNHS T , CI s , and o t he r s t a t i s t i ca l p r ocedur es a r e no ti n h e r e n t i n t h e s e p r o c e d u r e s b u t i n s t e a d s t e m f r o m


11/12

L O G I C A N D T H E N U L L 1 71i g n o r a n c e o f p r o p e r a p p l i c a t i o n s o f t h e s e t e c h n i q u e s .B y r e p l a c i n g N H S T w i t h C I s, w e a c tu a l l y d o h a r m b yg i v i n g t h e i l l u s i o n t h a t t h e p r o b l e m s a r e s o l v e d w h e n ,i n f a c t , t h e y h a v e n o t e v e n b e e n a d d r e s s e d .

I t i s a l s o w o r t h n o t i n g t h a t p r o b l e m s a s s o c i a t e dw i t h t h e i d e n t i f i c a t i o n o f a p p r o p r i a t e d e c i s i o n c r i t e ri aa r e l i k e l y t o p e r s i s t s i m p l y b e c a u s e o f t h e t y p e s o fq u e s t i o n s t h a t w e a s k . I n th e b e h a v i o r a l s c i e n c e s , w eh a v e t r a d i t i o n a l l y a s k e d y e s - n o q u e s t i o n s , D o e s Aa f f e c t B ? D o e s g o a l s e t ti n g a f f e c t p e r f o r m a n c e ? R e a -s o n a n d p r e v i o u s w o r k g i v e u s p r i o r b a se s f o r e x p e c t -i n g c e r t a i n a n s w e r s t o t h e s e q u e s t io n s . I f s ig n i f i c a n c et e s ts c o n t r i b u t e s u p p o r t f o r t h e s e e x p e c t a t i o n s , t h e nw e h a v e a t t ac k e d t h e p r o b l e m , a n d r e c e i v e d c o r r o b o -r a t i o n , f r o m b o t h t h e t h e o r e t i c a l a n d e m p i r i c a l s i d e s .U n f o r t u n a t e l y , o u r t h e o r i e s a r e r a r e l y p r e c i s e e n o u g ht o a l l o w f o r p r e d i c t i o n s o f p a r a m e t e r v a l u e s . T h u s ,o n l y e m p i r i c a l s u p p o r t f o r a g i v e n p a r a m e t e r v a l u e i sp o s s i b l e . O n t h e o t h e r h a n d , i f w e k e e p o u r q u e s t i o n sm o r e g e n e r a l , m o r e c o n s e r v a t i v e , t h e n t h e p o s s i b i l i t yo f h a v i n g b o t h t h e o r et i c al a n d e m p i r i c a l s u p p o r t e x -i s t s . A s a l w a y s , w e m u s t a p p l y t h e s a m e c a r e a n dt h o u g h t t o t h e i n t e rp r e t a t i o n o f e m p i r i c a l e v i d e n c et h a t w e a p p l y t o t h e o r e t i c a l e v i d e n c e .

F i n a l l y , l e t u s n o t f o r g e t t h a t j u d g m e n t i s r e q u i r e di n e v e r y a n a l y s i s o f s c i e n ti f i c i n f o r m a t i o n . T h e a b u s e so f N H S T h a v e c o m e a b o u t la r g el y b e c a u se o f a l ac ko f j u d g m e n t o r e d u c a t i o n w i t h r e s p e c t t o t h o se u s i n gt h e p r o c e d u r e . T h e c u r e l i e s i n i m p r o v i n g e d u c a t i o na n d , c o n s e q u e n t l y , j u d g m e n t , n o t i n a b o l i s h i n g t h em e t h o d . M i n d l e s s a p p l i c a ti o n o f a n y p r o c e d u r e c a u s e sp r o b l e m s , a n d d i s c a r d i n g a p r o c e d u r e b e c a u s e i t h a sb e e n m i s a p p l i e d e n s u r e s t h e p r o v e r b i a l l o s s o f b o t hb a b y a n d b a t h w a t e r .

e f e r en c e s

Bakan, D. (1966). The test of significance in psych olog icalresearch. Psychological Bul l e t i n , 66 , 1-29.

Barrick, M. R. , Mo unt, M. K. (1991). The big five per-s ona l i t y d i me ns i ons a nd j ob pe r f o r ma nc e : A me t a -analysis. Personnel Psychology , 44 , 1-26.

Carver, R.P. (1978). The case against statist ical signifi-canc e testing. H a r v a r d E d u c a t i o n a l R e v i e w , 4 8 , 378- 399 .

Cohen, J. (1990). Things I have learned (so far) . A m e r i c a nPsychologi s t , 45 , 1304-1312.

Cohen, J. (1994). The Earth is round (p < .05). A m e r i c a nPsychologi s t , 49 , 997- 1003 .

Cortina, J. M., DeS hon , R. P. (1997). Ext reme groups vs .o b s e r v a t io n a l d e s i g n s : I s s u e s o f a p p r o p r i a t e n e s s f o r t h e

detect ion of interact ions . Manu scr ip t submit ted for pub-lication.

Dellow , E. L. (1970). M e t h o d s o f s c i e n ce . Ne w Yor k : Un i -verse Books .

Doo ling, D. J. , Dan ks, J. H. (1975). Go ing bey ond testsof s ignif icance: I s psych ology ready? B u l l e ti n o f t h e P s y -chonomic Socie t y , 5 , 15-17.

Frick, R. W . (1995). A ccep ting the null hypothesis. M e m o r yCognition, 23, 132-138.

Gigerenzer, G. (1993). The superego, the ego, and the id instatist ical reasoning. In G. Keren C. Lew is (Eds.), Ahandb ook fo r data analys i s i n t he behavioral s c i ences :M e t h o d o l o g i c a l i s s u e s (pp. 311-339) . Hi l l sdale , NJ:Erlbaum.

Harr i s , R. J . (1994) . A N O V A : A n a n a l y s i s o f v a r i a n c epr imer . I tasca , IL: F . E . Peacock.

Hunter, J . E. , Schm idt, F. L . (1990). M e t h o d s o f m e t a -analys i s : Co rrect ing er ror and b ias i n r esearch f i ndings .Newbury Park , CA: Sage .

Kant, I. (1781). Cr i t ique o fpure r eason. Riga, Latvia : Fr ie-drich Hartknoch.

Keynes, J. M. (1921). A t r eat i s e on probabi l i t y . London :MacMillan.

Kuhn, T. (1962). The s t ructure o f s c i ent i f i c r evolu t ions .Chicago : Univers i ty of C hicago Press .

Laka tos, I . (1978). Falsification and me thod olog y of scien-tific research programs. In J. W orrall G. Currie (Eds.) ,T h e m e t h o d o l o g y o f s c i e n ti f ic r e s e a r c h p ro g r a m s : I m r el a k a to s p h i l o s o p h i c a l p a p e r s (Vol. 1). Cambridge, En-gland: C ambridge Univers i ty Press .

Meehl , P . (1967). Theory- tes t ing in psychology and phys-ics: A methodological paradox. P h i l o s o p h y o f S c i e n c e ,34, 103-115.

Mill, J. S. (1872). A sys t em of l ogic: Rat ioc inat i ve an d in-duct i ve (8th ed.) . Lon don : Lon gm ans, Green, Reader, andDyer .

Miller, R. W. (1979). Reason and commitment in the socialsciences. Phi losophy and Publ i c Af fa i r s , 6 , 241- 266 .

Miller, R. W. (1987). Fac t and method: Explanat ion , con-f i rmat ion , and r eal i t y i n t he natural and the socia l s c i -ences . Princeton, NJ: Princeto n University Press.Nant re l la , M.G. (1972) . The re la t ion between conf idenceintervals and tests of significance. In R. Kirk (Ed.), Sta-t i s ti ca l i ssues : A r eader fo r t he beha vioral s c i ences (pp.113-117 . Monterey , CA: Brooks/Cole .

Ne ym an, J. , Pearson , E. S. (1928). On the use and inter-pretation of certain test cri teria for the purposes of sta-tistical inference. Biomet r i ka , 20A, 175-263.

Ne ym an, J. , Pearson , E. S. (1933). On the testing of sta-tistical hyp oth ese s in relation to probabilities a priori.


12/12

72 CORTINA AND DUNLAPProceedings o f the Cambridge Philosoph ical Society 29492.

Popper, K. R. (1959). The logic o f scientific discovery. NewYork: Basic Books.

Putnam, (1981). Reason truth an d history. Cambridge,MA: Harvard University Press.

Rozeboom, W. W. (1960). The fallacy of the null hypoth-esis significance test. Psycholog ical Bulletin 57 416-428.

Schmidt, F.L. (1996). Statistical significance testing andcumulative knowledge in psychology: Implications fortraining of researchers. Psychological Methods 1, 115-129.

Serlin, R. C., Lapsley, D. K. (1985). Rationality in psy-chological research: The good-enough principle. Ameri-can Psychologist 40 73-83.

Thompson , B. (1992). Two and one-h alf decades of lead-ership in measurement and evaluation. Journal of Coun-seling and Development 70 434-438.

Rec eiv ed July 13, 1996Revision received Novemb er 25, 1996

Accep ted Decembe r 17, 1996

MERIC N PSYCHOLOGIC L SSOCI TIONSU BS C R I P TI O N C L A I M S I N F O R M A T I O N Toaay s Date:We provide this form to assist members, institutions, and nonmember iadividuals with any subscription problem. With the appropriate informationwe can begin a resolution. If you use the services of an agent, please do NOT duplicate claims through them and directly to us. PLEASE PRINTCLEARLY AND IN INK IF POSSIBLE.PRINT FULLNOME OR KEY NAMEOF INSTITUTIONADDRESS

CITY STATFJCOUNTRY ZIPYOUR NAMEAND PHONE NUMI~ERTITLE

MEMBER OR CUSTOMERNUMBER(MAY BE FOUND ON ANY PAST ISSUE LABEL)DATE YOURORDER WAS MAILED OR PHONED)

PREPAID C HE C K CHARGECHECK/CARD CLEAREDDATE:(If possible,~nd a copy, rontand back, of your cancelledcheck o help us in ourresearchof your claim.) ISSUES: MI SS I N G DAMAGEDVOLUMEOR YEAR NUM E R OR MONT H

DATE RECEIVED:ACTION TAKEN:STAFF NAME:

(TO BE FILLED OUT BY APA STAFF)DATE OF ACTION:INV. NO. DATE:LABEL NO. DATE:

Send this form to APA Subscription Claims, 750 First Street, NE, Washington, DC 20002-4242or FAX a copy to (202) 336-5568.

PLEASE DO NOT REMOVE. A PHOTOCOPY MAY BE USED.

logic and purpose significance testing

Documents