least squares problems from wikipedia, the free encyclopedia the method of least squares is a...

12
Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard approach to the approximate solution of overdetermined systems, i.e., more equations than unknowns. The most important application is in data fitting The least-squares method was first described by Carl Friedrich Gauss around 1794 Legendre was the first to publish the method, however.

Upload: magnus-johnston

Post on 20-Jan-2018

219 views

Category:

Documents


0 download

DESCRIPTION

The Problem Range of A x Ax b

TRANSCRIPT

Page 1: Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard…

Least Squares Problems

From Wikipedia, the free encyclopedia

The method of least squares is a standard approach to the approximate solution of overdetermined systems, i.e., more equations than unknowns.

The most important application is in data fitting

The least-squares method was first described by Carl Friedrich Gauss around 1794

Legendre was the first to publish the method, however.

Page 2: Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard…

The Problem:

nmnmAbAx , is ,

residual theis

,min

:such that Find

22

m

n

Axbr

yb-AyAxb

x

Page 3: Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard…

The Problem

Range of A

xAx

b

Page 4: Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard…

If we have 21 data points we can find a unique polynomial interpolant to these points by solving:

Data-Fitting

20,,0 ),()( ixfxP ii

20

0

20,,0 ),(j

ijii ixfxc

)(

)(

1

1

20

0

20

0

202020

2000

xf

xf

c

c

xx

xx

Page 5: Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard…

Without changing the data points we can do better by reducing the degree of the polynomial

In the previous example: Polynomial of degree 8:

Polynomial Least Squares Fitting

)(

)(

1

1

20

0

8

0

82020

800

xf

xf

c

c

xx

xx

Page 6: Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard…

Orthogonal Projection and the Normal Equations

Theorem:

ly equivalentor ly equivalent

)range(

min satisfies A vector

given. be and )(Let

22

AxPbbAAxA

Ar

AwbAxbx

bnmA

TT

w

n

mnm

n

Page 7: Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard…

Pseudoinverse

mnTT AAAA 1)(

)( 1AAT exists If A has full rank then

Is called the Pseudoinverse, and

bAx Is the least squares solution

Page 8: Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard…

Four Algorithms

1. Find the Pseudoinverse

2. Solve the Normal Equation (A full rank):

TT AAAA 1)(

Then calculate bAx

Requires A to have full rank

AAT Is positive definite and we use the Cholesky factorization

Page 9: Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard…

Four Algorithms

3- QR Factorization:^^RQA Reduced QR

^^TQQP Orthogonal projector onto range(A)

)(Range ^^

AbQQPby T

hence and such that then yAxx

bQxR

Q

bQQxR

T

T

T

^^

^

^^^^

by multipl-left

Q

Page 10: Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard…
Page 11: Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard…

Four Algorithms

4- SVD TVUA^^ Reduced SVD

^^TUUP Orthogonal projector onto range(A)

)(Range ^^

AbUUPby T

hence and such that then yAxx

bUxV

U

bUUxVU

TT

T

TT

^^

^

^^^^

by multipl-left

Page 12: Least Squares Problems From Wikipedia, the free encyclopedia The method of least squares is a standard…