7/25/2019 Multinominal logit regression Bohning.pdf

1/4

Ann. Inst . S ta t i s t . Math .

Vol. 44, No. 1, 197-200 (1992)

M U L T I N O M I A L L O G I S T I C R E G R E SS IO N A L G O R I T H M * *

D NKM R BI~HNING

Dep ar tm ent o f Epidernio logy, Free Univers i ty Ber l in , Augus tas tr . 37

1000 Ber l in ~5, Germany

(Received Ju ly 23, 1990; revised O ctob er 12, 1990)

A b s t r a c t . T h e l ow e r b o u n d p r i nc ip l e ( i n tr o d u c e d i n B S h n i n g a n d L i nd s a y

(1988, A n n . I n s t . S t a t i s t . M a t h . , 4 0 , 6 4 1 - 6 6 3 ) , B S h n i n g ( 1 9 8 9 , B i o m e t r i k a , 7 6 ,

3 7 5 - 3 8 3 ) c o n s i s t s o f r e p l a c i n g t h e s e c o n d d e r i v a t i v e m a t r i x b y a g l o b a l lo w e r

b o u n d i n t h e L o e w n e r o rd e r in g . T h i s b o u n d is u s e d in th e N e w t o n - R a p h s o n

i t e r a t io n i n s t e a d o f t h e H e s s i a n m a t r i x l e a d i n g t o a m o n o t o n i c a l l y c o n v e rg i n g

s e q u e n c e o f i t e r a t e s . H e r e , w e a p p l y t h i s p r i n c i p l e t o t h e m u l t i n o m i a l l o g is t ic

r e g r e s s i o n m o de l~ w h e r e i t b e c o m e s s p e c i f ic a l l y a t t r a c t i v e .

K e y w o r d s a n d p h r a s e s : K r o n e c k e r p r o d u c t , L o e w n e r o r d e r i n g , l o w e r b o u n d

p r i n c i p l e , m o n o t o n i c i t y .

i .

ntroduction

L e t L ( ~r ) d e n o t e t h e l o g - li k e li h o o d, V L ( n ) t h e s c o r e v e c t o r a n d V 2 L ( n ) t h e

s e c o n d d e r i v a t i v e m a t r i x a t r E R m . S u p p o s e

( 1. 1) V 2 L ( r ) > B

f o r a ll ~ a n d s o m e n e g a t i v e d e f i n it e m x m m a t r i x B . H e r e C > D d e n o t e s L o e w n e r

o r d e r in g o f t w o m a t r i c e s a n d m e a n s t h a t C - D i s n o n - n e g a t i v e d e fi n i te . C o n s i d e r

t h e s e c o n d o r d e r T a y l o r s e r ie s f o r t h e l o g - l ik e l i h o o d a t n 0 :

L n ) - L n o ) = n - n o ) T V L n o ) + ~ n - n o ) T V 2 L ( n o + a ( n - n o ) ) ( ~ - n o )

1 n 0 ) r B ~ n 0 )

> n - ~ 0 ) ~ V L ~ 0 ) + ~ ~ -

w h e r e w e h a v e u s e d ( 1 .1 ) to a c h i e ve t h e l o w e r b o u n d f o r L . M a x i m i z i n g t h e

r i g h t - h a n d s i de o f t h e a b o v e i n e q u a l i t y y i e ld s t h e

L o w e r B o u n d

iterate nLB -----

lro - - B - 1 V L ( n o ) . W e h a v e t h e f ol lo w i ng :

* Supplem ent to M onotonici ty of quadrat ic-app roxima t ion a lgor i thms by B6hning and

Lindsay (1988).

Ann. Ins t . S ta t is t . Math . ,

40, 641-663.

** This research was suppor ted by the German Research Foundation.

197

7/25/2019 Multinominal logit regression Bohning.pdf

2/4

9 8 D A N K M A R B O H N I N G

T H E O R E M 1 . 1. ( B S h n i n g a n d L i n d s a y ( 1 9 8 8 ) ) ( i) M o n o t o n i c i t y ) F o r t h e

L o w e r B o u n d i t e ra t e w e h av e

L ( ~ L B ) k L ( ~ 0 ) w i t h > i f ~LB ~o .

(ii) C o n v e r g e n c e ) L e t r j ) b e a s e q u e n c e c r ea t e d b y t h e lo w e r b o u n d a l g o r i th m .

I f L i s b o u n d e d a b o v e i n a d d i t i o n , t h e n

I I V L ~ y ) l l . , o .

3 -.-~.x:

2 M u l t in o m i a l lo g i s t ic r e g r e s s io n

W e o b s e rv e v e c t o rs Y = ( Y l , . . . ,

Y k + l ) T ,

w i t h y~ = 0 f o r a l l i b e s i d e s o n e j

w i t h

y j

= 1 a n d c o r r e s p o n d i n g p r o b a b i li t y p j , i m p l y i n g

E Y = p , C o v Y = A p - p p T , A p =

0

0

P k + l

R e c a l l t h a t t h e m u l t i n o m i a l l o g i t - m o d e l i s g i v e n b y

k

j = l

k

j = l

f o r i - - - 1 , . . . , k ,

w h e r e x = ( x l , . . .

,Xm T

i s t h e v e c t o r o f c o v a r i a te s , a n d v ( i) i s t h e p a r a m e t e r

v e c t o r c o r r e s p o n d i n g t o t h e i - t h r e s p o n s e c a te g o r y F o r r e a s o n s o f s i m p l i c i t y in

p r e s e n t a t i o n , c o n s i d e r t h e l o g - li k e l ih o o d o f j u s t o n e o b s e r v a t i o n Y :

k k ]

lo g H p ~ J = E Y j l r ( J ) r x - lo g 1 + E e x p ( + ( J ) T : ) .

j = l j = l j = l

L et 7r = ( r ~ l ) , . . . , ~(1),,m,

, 7r~ ) , . . . , k))T,,m

d e n o t e t h e m k - v e c t o r o f

m k

p a -

r a m e t e r s , t h e u p p e r i n d e x g o i n g a l o n g w i t h t h e r e s p o n s e c a te g o ry , t h e l ow e r i n d e x

w i t h t h e c o v a r i a te . W e h a v e fo r t h e p a r t i a l d e r i v a t i v e

O L e x p ( r (h )T ~ )

:97r gh = yh xg - 1 + ~ jk= l ex p w J ) Tx ) xg = Yh - - ph )X g

w i t h t h e n o t a t i o n P h --- e x p ( ~ ( h ) r x ) / ( 1 + ~ k = l e x p ( ~r j) T x ) ). T h i s y i e l d s t h e s c o r e

v e c t o r

V L T r ) = [ y l - i 5 1 ) x l , . . . , y l - ~ l ) x m , . . . , Y k - - ~ k ) X l , . . . , Y k - - ~ k ) X m ] T

---- Y - p ) x

7/25/2019 Multinominal logit regression Bohning.pdf

3/4

MULTINOMIAL LOGISTIC REGRESSION ALGORITHM 199

where @ is the K ronecker product A

@

B of two arbitra ry matrices. T he observed

information

can

be easily computed to be

leading to t he observed information matrix

T he proof of th e following lemm a is straightforw ard.

LEMMA

.1.

If A B the n for s ymm etric , nonnegative definite C:

LEMMA

.2.

p

pT