notes on multiple regression using matrices multiple regression tony e. smith ese 502: spatial data...
Post on 21-Dec-2015
238 views
TRANSCRIPT
NOTES ON MULTIPLE REGRESSION USING MATRICES
Multiple Regression
Tony E. Smith
ESE 502: Spatial Data Analysis
Matrix Formulation of Regression
Applications to Regression Analysis
SIMPLE LINEAR MODEL
Data: ( , ) , 1,..,i iy x i n
Parameters: 2
0 1( , ) ,
Model:
0 1 , 1,..,i i iY x i n 2~ (0, ) , 1,..,i iid N i n
0 1( | ) , 1,..,i i iE Y x x i n
SIMPLE REGRESSION ESTIMATION
Data Points: ( , )i iy x
Predicted Value:
0 1ˆ ˆˆi iy x
Estimate Conditional Mean:
0 1( | )E Y x x
iyy
ˆiy
iyix
Line of Best Fit
where:
0 1
20 1 0 11( , )
ˆ ˆ( , ) min [ ( )]n
i iiy x
STANDARD LINEAR MODEL
Data: 1( , ,.., ) , 1,..,i i iky x x i n
Parameters: 2
0 1( , ,.., ) ,k
Model:
0 1, 1,..,
k
i j ij ijY x i n
2~ (0, ) , 1,..,i iid N i n
1 0 1( | ,.., )
k
i i ik j ijjE Y x x x
STANDARD LINEAR MODEL (k = 2)
Data: 1 2( , , ) , 1,..,i i iy x x i n
Parameters: 2
0 1 2( , , ) ,
Model:
0 1 1 2 2 , 1,..,i i i iY x x i n 2~ (0, ) , 1,..,i iid N i n
1 2 0 1 1 2 2( | , )i i i i iE Y x x x x
REGRESSION ESTIMATION (for k =2)
Plane of Best Fit
1 2( , )i ix x
iy
ˆiy
y
1x
2x
Data Points: 1 2( , , )i i iy x x
Predicted Value:
0 1 1 2 2ˆ ˆ ˆˆi i iy x x
where:
0 1 2
20 1 2 0 1 1 2 21( , , )
ˆ ˆ ˆ( , , ) min [ ( )]n
i i iiy x x
MATRIX REPRESENTATION OFTHE STANDARD LINEAR MODEL
Vectors and Matrices:
1
2 ,:n
YYY
Y
11 12
21 22
1 2
11 ,: : :1 n n
x xx xX
x x
0
1
2
,
1
2:n
Matrix Reformulation of the Model:
00where: 0
0
1 0 00 1and 00 0 1
nI
Y X 2~ (0, )nN I
LINEAR TRANSFORMATIONSIN ONE DIMENSION
Linear Function: ( )f x a x
(1) 1f a a
( ) (1)f x f x
Graphic Depiction:
0 1 a x
a x
LINEAR TRANSFORMATIONSIN TWO DIMENSIONS
Linear Transformation:
1 11 1 12 2
2 21 1 22 2( ) x a x a xf x f x a x a x
11 12
21 22
1 0,0 1a af fa a
11 2
2
1 00 1
xf f x f xx
SOME MATRIX CONVENTIONS
11 1 11 1
( ) ( )
1 1
: : : :n k
k n n k
k kn n kn
a a a aA A A A
a a a a
1
( 1) (1 ) 1: ( ,., )k k k
k
aa a a a a a
a
Transposes of Vectors and Matrices:
Symmetric (Square) Matrices: A A
Important Example: ( )A A A A i symmes tric
Column Representation of Matrices:
11 1 111
1
1 1
: : : ,.., : ( ,.., )n n
k
k kn k kn
a a aaA a a
a a a a
Row Representation of Matrices:
11 1 11 1 1
1 1
( ,.., ): : : :
( ,.., )
n n
k kn k kn k
a a a a aA
a a a a a
1 1 1: , : :k n
k n k
a x a xA A x Ax
a x a x
Matrix Multiplication:
1 1
1: , :
n
i ii
n n
a xa x a x a x x a
a x
Inner Product of Vectors:
1 1 1
1
1
,.., : :m
n m m
k k m
a b a bB B b b AB
a b a b
( )AB B A Transposes:
MATRIX REPRESENTATIONS OF LINEAR TRANSFORMATIONS
11 12
21 22
1 0,0 1a af fa a
11 2
2
1 0( ) 0 1xf x f f x f xx
For any Two-Dimensional Linear Transformation :
with :
11 1 12 2 11 12 1
21 1 22 2 21 22 2( ) a x a x a a xf x Axa x a x a a x
Graphical Depiction of Matrix Representation:
01
10
1
2
xx x
12
22
aa
11
21
aa
111
21
a xa
122
22
a xa
Ax
Inversion of Square Matrices (as Linear Transformations):
11 1 111
1 1
1 0: : : ,.., : : ,.., :
0 1
n n
n
n nn n nn
a a aaAI
a a a a
1111
1
1 0: ,.., : : ,.., :
0 1
n
n nn
aaA
a a
1 1nA A I AA
DETERMINANTS OF SQUARE MATRICES
11 12
21 22
a aA a a
det( )A
11 22 21 21a a a a
| det( ) |A
Area of the of the unit square undimage er A
12
22
aa
11
21
aa
NONSINGULAR SQUARE MATRICES
12
22
aa
11
21
aa
1A exists
11 21
21 22
a aanda a are not colinear
det 0A
A is nonsingular
LEAST-SQUARES ESTIMATION
2 20 1 11 1
( ) [ ( )] ( )n n
i i k ik i ii iS y x x y x
2 2
1 1 12 ( )
n n n
i i i ii i iy y x x
General Sum-of-Squares:
General Regression Matrices:
11 11
2 21 2
1
11 ,: : : : :1
k
k
n n nk
x xxx x xX
x x x
0
1 ,:k
1
2 ,:n
yyy
y
1
1
,:i
i
ik
xx
x
1
2:n
xxX
x
( ) 2S y y y X X X
DIFFERENTIATION OF FUNCTIONS
General Derivative:
0
( ) ( )( ) limd
dxf x f x
f x
Example: 2( )f x x
2 2
0
( )( ) limd
dxx x
f x
2 2 2
0
( 2 )lim
x x x
0lim (2 )x
( ) 2o oddx f x x
( )of x
ox
( )of x
PARTIAL DERIVATIVES
z
1 2( , )z f x x
2ox
1 2( , )o ox x
1
1 2 1 21 2 0
( , ) ( , )( , ) lim
o o o oo o
xf x x f x x
f x x
VECTOR DERIVATIVES
1( ) ( ,.., )nf x f x x
11( )( )
( ) : :( ) ( )
n
x
n
x
x
f xf xf x
f x f x
Derivative Notation for:
1( ) ( ,.., ) , 1,..,ii nxf x f x x i n
Gradient Vector:
TWO IMPORTANT EXAMPLES
1( )
n
i iif x a x a x
Linear Functions:
( ) , 1,..,i if x a i n ( )x f x a
Quadratic Functions:
1
1( ) ( ,.., ) :n
n
a xf x x Ax x x
a x
1 1
n n
i ij ji jx a x
1( )
n
i iix a x
Quadratic Derivatives:
1 1( )
n n
k kh hk hf x x Ax x a x
1 1( )
n n
i ih h ki kh kf x a x a x
i ia x a x
1 1( ) : :x
n n
a x a xf x Ax A x
a x a x
( ) ( ) 2xA A x Ax Ax
Symmetric Case:
MINIMIZATION OF FUNCTIONS
First-Order Condition:
Example:
( *) 0ddx f x
2( ) 2f x a bx x
( ) 2 2ddx f x b x
0 ( *) 2 2 * *ddx f x b x x b
( )f x
*x
1ox
1x
2ox
z
TWO-DIMENSIONAL MINIMIZATION
1 2( , )z g x x
1 1 2( , ) 0o ox g x x
2 1 2( , ) 0o ox g x x
( ) 0ox g x
LEAST SQUARES ESTIMATION
Solution for: 0 1
ˆ ˆ ˆ ˆ( , ,.., )k
min ( ) 2( )S y y y X X X
ˆ ˆ0 ( ) 2 2S X y X X
ˆX X X y
1ˆ ( )X X X y det( ) 0if X X
NON-MATRIX VERSION (k = 2)
1 2 1 2( , , ) , 1,.., , ( , , )i i iy x x i n y x x sample means Data:
Beta Estimates:
21 2 2 1 21 1 1 1
1 2 21 2 1 21 1 1
ˆn n n n
i i i i i i ii i i i
n n n
i i i ii i i
y x x y x x x
x x x x
1 2 1 1 2 2( , , ) ( , , )i i i i i iy x x y y x x x x deviation form
21 1 1 1 21 1 1 1
2 2 21 2 1 21 1 1
ˆn n n n
i i i i i i ii i i i
n n n
i i i ii i i
y x x y x x x
x x x x
0 1 1 2 2ˆ ˆ ˆ , :y x x where
EXPECTED VALUES OF RANDOM MATRICES
Random Vectors and Matrices
11 1
1
: : :k
n k
k kn
Y YY Y
Y Y
1
1 : ,n
n
YY Y
Y
Expected Values:
11 1
1
( ) ( )( ) : : :
( ) ( )
k
k kn
E Y E YE Y
E Y E Y
1( )( ) : ,
( )n
E YE Y
E Y
EXPECTATIONS OF LINEARFUNCTIONS OF RANDOM VECTORS
Linear Combinations
1 1( ) ( ) ( )
n n
i i i ii ia Y aY E a Y a E Y a E Y
Linear Transformations
1 1 ( ): ( ) : ( )
( )n n
a Y a E YAY E AY AE Y
a Y a E Y
EXPECTATIONS OF LINEARFUNCTIONS OF RANDOM MATRICES
Left Multiplication
Right Multiplication (by symmetry of inner products):
1 1 1
1
: : :k
h n n k
h h k
a Y a YAY A Y
a Y a Y
1 1 1
1
( ) ( )( ) : : : ( )
( ) ( )
k
h h k
a E Y a E YE AY AE Y
a E Y a E Y
( ) ( )k n n hYB Y B E YB E Y B
COVARIANCE OF RANDOM VECTORS
Random Variables :
Random Vectors:
( ) , 1,..,i iE Y i n
cov( , ) [( )( )] , 1,..,i j ij i i j jY Y E Y Y i n
1( ) ( ,.., ) ,nE Y
11 1 1 1 1 1 1 1 1 1
1 1 1
[( )( )] [( )( )]cov( ) : : : : : :
[( )( )] [( )( )]
n
n nn n n n n n n
E Y Y E Y YY
E Y Y E Y Y
1 1 1 1 1 1 1 1 1 1 1 1
1 1
( )( ) ( )( ): : : : :
( )( ) ( )( )n n n n n n n n n n
Y Y Y Y Y YE E
Y Y Y Y Y Y
cov( ) [( )( ) ]Y E Y Y
COVARIANCE OF LINEARFUNCTIONS OF RANDOM VECTORS
Linear Combinations:
Linear Transformations:
cov( ) [( )( ) ]AY E AY A AY A
( )E Y
[ ( )( ) ]E A Y Y A
[( )( ) ]AE Y Y A
[( )( ) ]AE Y Y A ( Right Mult )
( Left Mult )
cov( ) cov( )AY A Y A
cov( ) cov( )a Y a Y a
TRANSLATIONS OF RANDOM VECTORS
Translation: Y b Y
cov( ) [( { })( { }) ]b Y E b Y b b Y b
Means: ( ) ( ) ( ) ( )E b Y E b E Y b E Y
Covariances:
( ) ( )E b AY b AE Y
( )E Y
[( )( ) ] cov( )E Y Y Y
cov( ) cov( )b AY A Y A
RESIDUAL VECTOR IN THE STANDARD LINEAR MODEL
Linear Model Assumption: 2~ (0, ) , 1,..,i iid N i n
Residual Means: ( ) 0 , 1,.., ( ) 0iE i n E
Residual Covariances:
2 2var( ) ( ) , cov( , ) ( ) 0 ,i i i j i jE E j i
cov( ) [( 0)( 0) ] ( )E E
21 1 1 1 1 1
21 1
( ) ( ) 0: : : : : :
( ) ( ) 0
n n
n n n n n n
E EE
E E
2cov( ) nI
MOMENTS OF BETA ESTIMATES
Linear Model: 2, ~ (0, )nY X N I
1 1ˆ ( ) ( ) ( )X X X Y X X X X 1 1( ) ( )X X X X X X X 1( )X X X
Mean of Beta Estimates:
1ˆ ˆ( ) ( ) ( ) ( )E X X X E E (Unbiased Estimator)
Covariance of Beta Estimates:
1 1ˆcov( ) cov[ ( ) ] cov[( ) ]V X X X X X X 1 1 2 1 1( ) cov( ) ( ) ( ) ( )X X X X X X X X X X X X
2 1ˆcov( ) ( )V X X
ESTIMATION OF RESIDUALVARIANCE
Residual Variance: 2 2var( ) ( ) , 1,..,i iE i n
Residual Estimates: ˆ ˆ , 1,..,i i iy y i n
Natural Estimate of Variance:
2 211
1 1ˆ ˆ ˆ ˆ ˆ ˆˆ , ( ,.., )n
i nin n where
Bias-Correct Estimate of Variance:
2 1( 1) ˆ ˆn ks (Compensates for Least Squares)
ˆ ˆ ˆˆ ˆ( ) ( ) ( )S y y y y