Ordinary Least-Squares

Ordinary Least-Squares

Ordinary Least-Squares

Outline

• Linear regression• Geometry of least-squares• Discussion of the Gauss-Markov

theorem

Ordinary Least-Squares

One-dimensional regression

a

b

Ordinary Least-Squares

One-dimensional regression

axb

Find a line that represent the

”best” linear relationship:

a

b

Ordinary Least-Squares

One-dimensional regression

xabe iii

• Problem: the data does not go through a line

xab ii

a

b

Ordinary Least-Squares

One-dimensional regression

xabe iii

• Problem: the data does not go through a line

• Find the line that minimizes the sum:

2)( i

ii xab

xab ii

a

b

Ordinary Least-Squares

One-dimensional regression

x̂

xabe iii

2)()( i

ii xabxe

• Problem: the data does not go through a line

• Find the line that minimizes the sum:

• We are looking for that minimizes

2)( i

ii xab

xab ii

a

b

Ordinary Least-Squares

Matrix notation

Using the following notations

and

na

a

:1

a

nb

b

:1

b

Ordinary Least-Squares

Matrix notation

Using the following notations

and

we can rewrite the error function using linear algebra as:

na

a

:1

a

nb

b

:1

b

2

2

)(

)()(

)()(

ab

abab

xxe

xx

xabxe

T

iii

Ordinary Least-Squares

Matrix notation

Using the following notations

and

we can rewrite the error function using linear algebra as:

na

a

:1

a

nb

b

:1

b

2

2

)(

)()(

)()(

ab

abab

xxe

xx

xabxe

T

iii

Ordinary Least-Squares

Multidimentional linear regression

Using a model with m parameters

j

jjmm xaxaxab ...11

Ordinary Least-Squares

Multidimentional linear regression

Using a model with m parameters

2a

j

jjmm xaxaxab ...11

1a

b

Ordinary Least-Squares

Multidimentional linear regression

Using a model with m parameters

2a

j

jjmm xaxaxab ...11

1a

b

Ordinary Least-Squares

Multidimentional linear regression

Using a model with m parameters

and n measurements

j

jjmm xaxaxab ...11

2

2

1,

2

1 1, )()(

Axb

b

x

m

jjji

n

i

m

jjjii

xa

xabe

Ordinary Least-Squares

Multidimentional linear regression

Using a model with m parameters

and n measurements

j

jjmm xaxaxab ...11

2

2

1,

2

1 1,

)(

)()(

Axbx

b

x

e

xa

xabe

m

jjji

n

i

m

jjjii

Ordinary Least-Squares

Axb

mmnn

m

n x

x

aa

aa

b

b

:.

..

::

..

:1

,1,

,11,11

Axb

Ordinary Least-Squares

Axb

)...(

:

)...(

:.

..

::

..

:

,11,

,111,11

1

,1,

,11,11

mmnnn

mm

mmnn

m

n

xaxab

xaxab

x

x

aa

aa

b

b

Axb

Ordinary Least-Squares

Axb

)...(

:

)...(

:.

..

::

..

:

,11,

,111,11

1

,1,

,11,11

mmnnn

mm

nmnn

m

n

xaxab

xaxab

x

x

aa

aa

b

b

Axb

parameter 1

Ordinary Least-Squares

Axb

)...(

:

)...(

:.

..

::

..

:

,11,

,111,11

1

,1,

,11,11

mmnnn

mm

nmnn

m

n

xaxab

xaxab

x

x

aa

aa

b

b

Axbmeasurement n

parameter 1

Ordinary Least-Squares

Minimizing )(xe

x x ifminimizes )e( min

Ordinary Least-Squares

Minimizing )(xe

x x ifminimizes )e( min

minx

)(xe

Ordinary Least-Squares

Minimizing )(xe

x x ifminimizes )e( min

is flat at

minx

)(xe

)e(x minx

Ordinary Least-Squares

Minimizing )(xe

x x ifminimizes )e( min

0x )( mineis flat at

minx

)(xe

)e(x minx

Ordinary Least-Squares

Minimizing )(xe

x x ifminimizes )e( min

0x )( mineis flat at

does not go downaround)e(x

minx

minx

)(xe

)e(x minx

Ordinary Least-Squares

Minimizing )(xe

x x ifminimizes )e( min

0x )( mine

definite-semi

positive is )( minxeH

is flat at

does not go downaround)e(x

minx

minx

)(xe

)e(x minx

Ordinary Least-Squares

Positive semi-definite

x0Axx

A

allfor ,

T

definite-semi positive is

In 1-D In 2-D

Ordinary Least-Squares

Minimizing )(xe

x̂

xxHx )ˆ(2

1e

T)(xe

Ordinary Least-Squares

Minimizing 2

)( Axbx e

x̂

xxHxx )ˆ(2

1)( e

Te

Ordinary Least-Squares

Minimizing

x x ifminimizes )e( ˆ

bAxAA TT ˆ

definite-semi

positive is AAT2

2)( Axbx e

Ordinary Least-Squares

Minimizing 2

)( Axbx e

x x ifminimizes )e( ˆ

bAxAA TT ˆ

definite-semi

positive is AAT2

Always true

Ordinary Least-Squares

Minimizing 2

)( Axbx e

x x ifminimizes )e( ˆ

bAxAA TT ˆ

definite-semi

positive is AAT2

Always true

The normal equation

Ordinary Least-Squares

Geometric interpretation

Ordinary Least-Squares

Geometric interpretation• b is a vector in Rn

b

Ordinary Least-Squares

Geometric interpretation• b is a vector in Rn

• The columns of A define a vector space range(A)

b

2a

1a

Ordinary Least-Squares

Geometric interpretation• b is a vector in Rn

• The columns of A define a vector space range(A)• Ax is an arbitrary vector in range(A)

b

2a

1a

Ax 2211 aa xx

Ordinary Least-Squares

Geometric interpretation• b is a vector in Rn

• The columns of A define a vector space range(A)• Ax is an arbitrary vector in range(A)

b

2a

1a

Axb

Ax 2211 aa xx

Ordinary Least-Squares

Geometric interpretation• is the orthogonal projection of b onto range(A)

b

2a

1a

xA ˆˆˆ 2211 aa xx

xAb ˆ

bAxAAxAbA TTT ˆˆ 0

xA ˆ

Ordinary Least-Squares

The normal equation: bAxAA TT ˆ

Ordinary Least-Squares

The normal equation:

• Existence: has always a solution

bAxAA TT ˆ

bAxAA TT ˆ

Ordinary Least-Squares

The normal equation: bAxAA TT ˆ

• Existence: has always a solution

• Uniqueness: the solution is unique if the columns

of A are linearly independent

bAxAA TT ˆ

Ordinary Least-Squares

The normal equation:

b

2a1a

xA ˆ

• Existence: has always a solution

• Uniqueness: the solution is unique if the columns

of A are linearly independent

bAxAA TT ˆ

bAxAA TT ˆ

Ordinary Least-Squares

Under-constrained problem

2a

1a

b

Ordinary Least-Squares

Under-constrained problem

2a

1a

b

Ordinary Least-Squares

Under-constrained problem

2a

1a

b

Ordinary Least-Squares

Under-constrained problem

• Poorly selected data• One or more of the

parameters are redundant

2a

1a

b

Ordinary Least-Squares

Under-constrained problem

• Poorly selected data• One or more of the

parameters are redundant

Add constraints

x bAAxA xmin with TT 2a

1a

b

Ordinary Least-Squares

How good is the least-squares criteria?• Optimality: the Gauss-Markov theorem

Ordinary Least-Squares

How good is the least-squares criteria?• Optimality: the Gauss-Markov theorem

Let and be two sets of random variables and define:

mmiiii xaxabe ,11, ...

ib jx

Ordinary Least-Squares

How good is the least-squares criteria?• Optimality: the Gauss-Markov theorem

Let and be two sets of random variables and define:

If

mmiiii xaxabe ,11, ...

ib jx

, ,0cov

, ,var

, ,0

:

ji),e(e

i)(e

i)E(e

a

ji

i

i

i,j

and allfor :A4

allfor :A3

allfor :A2

, variablesrandomnot areA1

Ordinary Least-Squares

How good is the least-squares criteria?• Optimality: the Gauss-Markov theorem

Let and be two sets of random variables and define:

If

Then is the best unbiased linear estimator

mmiiii xaxabe ,11, ...

ib jx

, ,0),cov(

, ,var

, ,0

:

jiee

i)(e

i)E(e

a

ji

i

i

i,j

and allfor :A4

allfor :A3

allfor :A2

, variablesrandomnot areA1

2minargˆiexx

Ordinary Least-Squares

a

b

ei

no errors in ai

Ordinary Least-Squares

a

b

ei

a

b

ei

no errors in ai errors in ai

Ordinary Least-Squares

a

b

homogeneous errors

Ordinary Least-Squares

a

b

a

b

homogeneous errors non-homogeneous errors

Ordinary Least-Squares

a

b

no outliers

Ordinary Least-Squares

a

b

a

b

no outliers outliers

outliers