MAT 2401Linear Algebra

2.5 Applications of Matrix Operations

http://myhome.spu.edu/lauw

HW Written Homework

Preview We will only focus on one

application – The Method of Least Squares.

Linear RegressionSuppose that a scientist has reason to believe that 2 quantities x and y are related linearly, that is,

y=mx+b.The scientist performs an experiment and collect data points (x1,y1),…,(xn,yn).

Linear Regressiony

x

,i ix y y mx b

ie

i i ie y mx b

Goals Find a line y=mx+b that minimize

the sum of the squares of the errors ei.

Use y=mx+b to estimate the function values.

Linear Regressiony

x

,i ix y y mx b

ie

i i ie y mx b

Matrix Equation

i i ie y mx b

Matrix Equation

1 11

2 22

11 Let , , ,

1 nn n

y exy ex b

Y X A Em

xy e

Matrix Form of Linear RegressionFor the linear regression model , the coefficients of the least squares regression line are given by

A= (XTX)-1XTYand the sum of squared error is

ETE1 11

2 22

11 Let , , ,

1 nn n

y exy ex b

Y X A Em

xy e

Plan… Computational Example HW Why the formula is correct? Very

Educational; Focus on the Ideas

Example 1Find the least squares regression line for the points (1,1), (2,2), (3,4), and (5,6).

1 11

2 22

1

11 Let , , ,

1

Then,

nn n

T T

y exy ex b

Y X A Em

xy e

A X X X Y

Example 1

Example 1

9 27 7

y x

Why? Give you some ideas why the

formula actually work.

Recall Q: How to find the minimum of a

function f(x)? A:

Recall Q: How to find the minimum of a

function f(x)? A: Q: How to find the minimum of a

function f(x,y)?

Recall: Sigma Notation A “compact” notation for sums to

avoid “…”30

2 2 2 2 2

1

1 2 3 30k

k

Recall: Sigma Notation

1 21

n

i ni

x x x x

Final value (upper limit)

Initial value (lower limit)Index

Recall: Linear Property 1

1 2

1 2

1

1

5

5

5 5 5

5

n

ii

n

ii

n

n

x x x

x

x

xx x

Recall: Linear Property 2

1 1 2 2

1 2 1

1

1 1

2

n n

n

n

i ii

n n

i ii i

n

x y x y x y

x x x y y y

x y

x y

Why? Let g(b,m) be the function of the

sum of the squared errors. We can find the critical point by

solving the equations0 and 0g g

b m

Why? Let g(b,m) be the function of the

sum of the squared error. We can find the critical point by

solving the equations

It can be shown that the critical point is a minimum (skip)

0 and 0g gb m

Why?

0 and 0g gb m

2

1

2

1

2

1

( , )n

ii

n

i ii

n

i ii

g b m e

y mx b

y mx b