chapter five orthogonality why orthogonal? least square problem accuracy of numerical computation

CHAPTER FIVECHAPTER FIVEOrthogonalityOrthogonality

Why orthogonal?Least square problemAccuracy of Numerical computation

Least square problemLeast square problem

Motivation: Curve fittingProblem formulation:

Given ,

Find such that

nmA mb

0x

xAbxAbnx

o

min

OutlinesOutlines

Results ofOrthogonal subspaceInner product spaces Least square problemsOrthonormal setsGram-Schmidt Orthogonalization process

n

The scalar product inThe scalar product in

Def: Let , .the

inner product (or scalar product)

of and is defined to be

The norm of is defined as

n

ny x

x y

n

iii yxyxyxyx

1

,

xxx ,

x

Theorem5.1.1: Let . Then

pf: By the law of cosines,

Note: If is the angle between , then

Thus

Def:

nyx &

cosyxyx

cos2222

yxyxxy

222

2

1cos xyyxyx

222

2

1iiii xyyx

ii yx yx

yx&

y

y

x

x

cos 11

yx

yx

0 yxyx

Cor: (Cauchy-Schwartz Inequality)

Let . Then

Moreover , equality holds

for some

nyx &

yx

.yxyx

Scalar and Vector ProjectionsScalar and Vector ProjectionsLet . Then the quantify is the scalar projection of onto and the vector is called the vector projection of ontoIf ,then the scalar projection of onto is and the vector projection of onto is

cosuxux

1u

x

x

x

u

u

uuxu

x

x y

y

1y

y

yx

yyy

yx

y

y

y

yx

Example: Find the point

on the line that

is closest to the point

(1,4)

Sol: Note that the vector is on the line

Then the desired point is

xy3

1

1

3w xy

3

1

7.0

1.2w

ww

wv

v

Q

xy3

1

4.1

Example: Find the equation of the plane

passing through and

normal to

Sol:

4,3,2

3,1,2

0

3

1

2

4

3

2

z

y

x

0341322 zyx

Example: Find the distance form

to the plane

Sol: a normal vector to

the plane is

desired distance

0,0,2P

022 zyx

2

2

1

3

2

v

vP

v

P

Orthogonal SubspaceOrthogonal Subspace

Def: Let be two subspace

of . We say that if

, and .Example: Let

but does not orthogonal to

YX &

n YX

0 yx Xx Yy

},{ 1espanX },{ 2espanY },{ 32 eespanZ

.& ZXYX

Y Z

Def: Let be a subspace of .

Then the orthogonal complement

of is defined as

Example: In ,

Y n

Y

},0{ FyyxxY n

3 },{}{ 321 eespanYespanY

}{},{ 321 espanXeespanX

Lemma: (i) Two subspaces

(ii) If is a subspace of ,then is

also a subspace of .

Pf: (i) If

(ii) Let

and

}0{ YXYX

Xn

Xn

YXx

002

xxxx

Xyy21

&

Xx

xyxyxyy 2121

0

Four Fundamental SubspacesFour Fundamental SubspacesLet

for some

for some

It will be shown later that

nmA

mnAor :xAxmnA

AN nn xAx 0

AN

AR

mm xAx 0

xAbb m { mnx }

AR xAbb n { nmx }

ARAN ARAN

ARANn

ARANm

Theorem5.1.2: Let . Then and pf: Let and _____(1) for some _____(2)

________(3) Also, if _____(4)

Similarly,

nmA ARAN ARAN

ANx ARy

0:, xiA :,&

1

iAym

ii

0:,)1(

iAxyx i

ANARzA 0

ARAN)4)(3(

mi ,1

i

ARAN

ARAN

miiAZARz ,1,0:,

ARARAN

Example: Let

Clearly,

00

21

02

01AA

1

0spanAN

1

2spanAN

2

1spanAR

0

1spanAR

ARAN

ARAN

Theorem5.2.2: Let be a subspace of ,

then (i)

(ii) If is a basis for and

is a basis for ,then

is a basis for .

pf: If The result follows

Suppose . Let

and

nnSS dimdim

}{ 1 rxx }{ 1 nr xx

n

S

SS

nSS }0{}0{S

)()(

XrankrXrank

rnrxxX )( 1

SXR )(

)()(1.2.5 XNXRS

Theorem

rnXNSTheorem

4.6.3

)(dim)dim(

}{ 1 rxx

To show that is a basis for ,

It remains to show their independency.

Let . Then

Similarly,

This completes the proof

n

01

n

iii xc Sx

011

r

iii

n

iii xcxxcx

ricxc i

r

iii 1,00

1

Sy

01

n

riii

n

ii xcyxcy

nricxc i

n

riii ,1,00

1

}{ 1 nxx

Def: Let are two subspaces

of . We say that is a

direct sum of ,denoted by

, if each

can be written uniquely as

a sum , where Example: Let

Then

but

VU &

W W

VU &

VUW

Ww

vu VvUu &

1

0

0

,

1

0

0

0

1

0

,

0

1

0

0

0

1

spanWspanVspanU

,3 WU VU 3 VU 3

Theorem5.2.2: If is a subspace of ,

then

pf: By Theorem5.2.2,

To show uniqueness,

Suppose

where

nS SSn

SSn

.&,, SvSuvuxx n

2211 vuvux SvvSuu 2121 ,&,

SSvvuu 1221

}0{SS

1221 & vvuu

Theorem5.2.4: If is a subspace of ,

then

pf: Let

If

(Why?)

S nSS )(

rS )dim(

rSTheorem

)dim(2.2.5

0 yxSx Sy

Sx

SS

SS

Remark: Let . i.e. , Since

and

are bijections .

nmA mnA :

ARANn

ArankArank )( ArankAnullityn ArankAnullity

ARARA :

ARARA :

Let nmA nA :

mA :

m

n

A

A

A

A

AN

AN

AN

AN

bijection

bijection

Cor5.2.5:

Let and . Then

either

(i)

or

(ii)

pf: or

Note that

nmA mb

bxAx n

my0&0 byyA

)(ARb )(ARb

0)()()( byANyANARb

0&0 byyAy m

)()( Tn ANAR

)(AR

b

)( AN

Example: Let . Find

The basic idea is that the row space and the sol. of are invariant under row operation.Sol: (i) (Why?)

(ii) (Why?)

(iii) Similarly,

and

(iv) Clearly,

431

110

211

A )(),(),(),( ARANARAN

bxA

000

110

101

~ r

row

AA

1

1

0

1

0

1

)( spanAR

0&00 3231 xxxxxAr

1

1

1

)( spanAN

000

210

101

~row

A

2

1

0

1

0

1

)( spanAR

1

2

1

)( spanAN

)()(& ARANARAN

Example: Let

(i)

and

(ii) The mapping

is a bijection

and

(iv) What is the matrix representation for ?

23:0

0

3

0

0

2

A

0

1

0

0

0

1

1

0

03 spanspanARAN

2)( AR

)(:)(

ARARAAR

0

3

2

02

1

2

1

x

x

x

x

)(:1

)(

ARARAAR

03

12

1

2

1

2

1 y

y

y

y

)( ARA

Linear Product SpaceLinear Product Space

A tool to measure the

orthogonality of two vectors in

general vector space

Def: An inner product on a

vector space is a function

Satisfying the following conditions:

(i) with equality iff

(ii)

(iii)

V

)(:, orCFVV

0, xx 0x

xyyx ,,

zyzxzyx ,,,

Example: (i) Let

Then is an inner product of

(ii) Let , Then is an

inner product of

(iii) Let and then

is

an inner product of

(iv) Let , is a positive function and

are distinct real numbers. Then

is

an inner product of

.,10&, niwyx in

ii

n

ii yxwyx

1

, nij

n

jij

m

i

baBA

11

,nmBA ,nm

].,[)(,, 0 baCxwgf 0)( xw

b

a

dxxgxfxwgf )()()(,

].,[0 baC

ngp , )(xw

nxx 1

)()()(,1

ii

n

ii xgxPxwgp

n

Def: Let be an inner

product of a vector space

and .

we say

The length or norm of is

,V

Vvu ,

0; vuvuv

vvv ;

Theorem5.3.1: (The Pythagorean Law)

pf:

222vuvuvu

22vu

vuvuvu ,2

vvvuuu ,,2,

u

v

vu

Example: Consider with inner product

(i)

(ii)

(iii)

(iv) (Pythagorean Law)

or

]1,1[0 C

1

1)()(, dxxgxfgf

xxdxx 101,11

1

212111,11

1 dx

32

3

2,

1

1 xxdxxxx

3

8

3

2211

222 xx

3

8)1(1,11

21

1

2 dxxxxx

Example: Consider with inner product

It can be shown that

(i)

(ii)

(iii)

Thus are

orthonormal set.

]1,1[0 C

dxxgxfgf )()(

1,

0sin,cos mxnx

mnnxmx cos,cos

mnnxmx sin,sin

Nnnxnx sin,cos

Example: Let

and let

Then not orthogonal

to

nmBA ,

AAA

baBA

F

ij

m

i

n

jij

,

,1 1

4

0

1

3

3

1

,

3

2

1

3

1

1

BA

ABA 6,

B6,5

FFBA

Def: Let be two vectors in an

inner product space . Then

the scalar projection of onto is

defined as

The vector projection of onto is

0& vu

V

u v

v

vu

v

vu

,,

u v

vvv

vu

v

vP

,

,

Lemma: Let be the vector projection

of onto . Then

for some

pf:

Pv &0u v

vkuPui

PPui

)(

)(

trivialii

PuP

vv

vu

vv

vu

PPuPPuPi

)(

0,

,

,

,

,,,)(

22

u

v

P

Pu

k

Theorem5.3.2: (Cauchy-Schwarz Inequality)

Let be two vectors in an

inner product space . Then

Moreover, equality holds are linear dependent.

pf: If

If

Equality holds

i.e., equality holds iff are linear dependent.

vu&

V

vuvu ,

vu&

thenv

trivialv

,0

,0

222

2

,

,PuuP

vv

vu TheoremnPythagorea

22vu

22222

, Puvvuvu

vvv

vuPu

,

,orv ,0

vu&

uPu

vP

Note:

From Cauchy-Schwarz Inequality.

This, we can define as the

angle between the two vectors

vu

vu

vu

vu

,cos

,0!

1,

1

.& vu

Def: Let be a vector space

A fun is said

to be a norm if it satisfies

with equality

scalar

V

:

vv

FV

0 v

wvwviii

vvii

vi

)(

,)(

0)(

Theorem5.3.3: If is an inner product

space, then

is a norm on

pf: trivial

Def: The distance between is defined

as vu

V

V

vu&

vvv

Example: Let . Then

is a norm

is a norm

is a norm for any

In particular ,

is the

euclidean norm

nx

Pn

i

P

iP

ini

xxiii

xxii

1

1

1

)(

)( max

n

iixxi

11

)(

1P

xxxxn

ii ,

1

2

2

Example: Let . Then

3

5

4

x

5

25

12

2

1

x

x

x

Example: Let

Thus,

However,

(Why?)

2

4&

2

121 xx

0, 21 xx2

221

2

22

2

225205 xxxx

1620

1642

21

2

2

2

1

xx

xx

Example: Let

Then

12

xxB

1B2B B

1 1 1

Least square problemLeast square problem A typical example:

Given

Find the best line

to fit the data .

or

or find such that

is minimum

Geometrical meaning :

niy

x

i

i ,1,

xccy 10

nn y

y

y

c

c

x

x

x

solve2

1

1

02

1

1

1

1

bxA

10 ,ccbxA

xccy 10

),( nn yx

),( 11 yx

Least square problem:

Given

then the equation

may not have solutions

The objective of least square problem is

trying to find such that

has minimum value

i.e., find satisfying

,& mnm bA

bxA

)()(.,. ARAColbei

x

xAb

x

xAbxAbnx

min )(AR

b

xA

Preview of the results:

It will be shown that

Moreover,

If columns of are Linear independent .

byPb

ARP

ARy

)(min

)(!

ybPbARy

)(min

bAxAA

xAbA

PbA

ANARPb

0

0

)()(

bAAAx

1

A

b

)(ARP

Theorem5.4.1: H. Let be a subspace of

C. (i)

for all

(ii)

pf:

where

If

Since the expression is unique,

result (i) is then proved .

(ii) follows directly from (i) by noting that

S m

Pbyb

SPb m

!,

}{\ PSy

SPbbybP

Symin

zPb

SSi m

)(

SzSP &}{\ PSy

2

0

2

2

2

yPPbyPPbybTheoremnPythogorea

SSz

SzPb

zPb

P

b

S

Question: How to find which solves

Answer: Let

From previous Theorem , we know that

normal equation

x

?min xAbbxAnx

nmA

0

0)(

)()(

xAAbA

PbA

ANARPb

)(AR

xAP

b

Theorem5.4.2: Let and

Then the normal equation . Has

a unique sol .

and is the unique least square

sol . to

pf: Clearly, is nonsingular (Why?)

is the unique sol. To normal equation .

is the unique sol . To the

least square problem (Why?)

( has linear independent columns)

nmA .)( nArank bAxAA

bAAAx

1

x

bxA AA

x

x

A

Note: The projection vector

is the element of that

is closet to in the least square

sense . Thus, The matrix

is called the

Projection matrix (that project any

vector of to )

bAAAAxAP

1

)(AR

b

AAAAP1

)(ARm

b

)(AR

P

Example: Suppose a spring obeys the Hook’s law

and a series of data are taken (with measurement

error) as

How to determine ?

sol: Note that

is inconsistent

The normal equation. is

The least square sol.

KxF

11

8

7

5

4

3

x

F

8

5

3

11

7

4

811

57

34

Kor

K

K

K

8

5

3

1174

11

7

4

1174 K

K

726.0K

Example: Given the data

Find the best least square fit by a linear function.sol: Let the desired linear function be The problem be comes to find the least square sol. of

Least square sol.

The best linear least square fit is

5

6

4

3

1

0

y

x

xccy 10

x

Ab

c

c

1

0

6

3

0

1

1

1

5

4

1

3

23

4

1

0 bAAAc

c

xy3

2

3

4

Example: Find the best quadratic least square

fit to the data

sol: Let the desired quadratic function . be

The problem becomes to find the least square

sol . of

least square sol .

the best quadratic least square fit is

25.0

25.0

75.2

2

1

0

c

c

c

4

3

4

2

2

1

3

0

y

x

2210 xcxccy

2

1

0

9

4

1

0

3

2

1

0

1

1

1

1

4

4

2

3

c

c

c

225.025.075.2 xxy

Orthonormal SetOrthonormal Set

Simplify the least square sol.

(avoid computing inverse)Numerical computational stability

Def: is said to be an orthogonal set in

an inner product space if

Moreover, if

, then

is said to be orthonormal

nvv 1

V

jiallforvv ji 0,

ijji vv ,

nvv 1

Example:

is an

orthogonal set but not

orthonormal

However ,

is orthonormal

1

5

4

,

3

1

2

,

1

1

1

321 vvv

42,

14,

33

32

21

1

vu

vu

vu

Theorem5.5.1: Let be an orthogonal

set of nonzero vectors in an inner

product space . Then they are

linear independent

pf: Suppose

is linear independent .

nvv 1

V

01

i

n

iiVc

nj

jjj

ijiiij

vv

njc

VVc

VVcVcV

1

,1,0

,

,0,

Example:

is an

orthonormal set of

with inner product

Note: Now you know the meaning what one

says that

Nnnsnx sincos,

2

1

,0 C

dxxgxfgf )()(

1,

xx sincos

Theorem5.5.2: Let be an

orthonormal basis for an inner

product space . If

then

pf:

nuu 1

V i

n

iiucv

1

vuc ii ,

jij

n

jj

ji

n

jjj

n

jjii

cc

uucucuvu

1

11

,,,

Cor: Let be an orthonormal

basis for an inner product space .

If and ,

then

pf:

V

nuu 1

n

iiiuau

1

n

iiiubv

1

n

iiibavu

1

,

n

iii

Theorem

n

iii

n

iii

ba

vua

vuavu

1

2.5.5

1

1

,

,,

Cor: (Parseval’s Formula)

If is an orthonormal

basis for an innerproduct space

and , then

pf: direct from previous corollary

nuu 1

V

n

iiiucv

1

n

iicv

1

2

Example:

and from

an orthonormal basis for .

If , then

and

2

12

1

1u

2

12

1

2u

22

2

1

x

xx

2, 21

2

xxux

,

2, 21

1

xxux

221

121

2.5.5

22u

xxu

xxx

Theorem

22

21

2

21

2

212

22xx

xxxxx

Example: Determine without

computing antiderivatives .

sol:

and is an orthonormal set of

xdx

4sin

xx

x

xxxxdx

2cos2

1

2

1

2

1

2

2cos1sin

sinsin,sinsin

2

22224

x2cos,2

1 ,0 C

4

3

2

1

2

1

sinsin

22

224

xxdx

Def: is said to be an orthogonal matrix if the column vectors of form an orthonormal set in Example: The rotational matrix

and the elementary reflection

matrix are orthogonal

matrix .

nmQ

nQ

cossin

sincos

cossin

sincos

Properties of orthogonal matrix:

at be orthogonal . Then

The column vectors of form an

orthonormal basis for

Preserve inner product

preserve norm

preserve angle .

nmQ

)(

)(

,,)(

)(

)(

)(

22

1

vi

xxQv

yxyQxQiv

QQiii

IQQii

Qin

Note: Let the columns of form

an orthonormal set of .Then

and the least

square sol to is

This avoid computing matrix inverse .

nA

IAA

bxA

bAbAAAx

1

Cor5.5.9:

Let be a nonzero subspace of

and is an orthonormal

basis for . If

then the projection of onto is

pf:

Kuu 1

S

S ,1 KuuU

m

SbP

bUUP

bUU

bUUUUP

Note: Let columns of be

an orthonormal set

The projection of onto

is the sum of the projection

of onto each .

KuuU 1

b

u

u

uubUU

K

K

1

1

i

K

ii ubu

1

iu

b

b

)(UR

Example: Let Find the vector in that is closet to

Sol: Clearly is a basis for . Let Thus

Hw: Try

What is ?

.4,3,5 w

yxyxS ,0,,P S

21,ee

S

0

1

0

0

0

1

U

0

3

5

4

3

5

000

010

001

wUUP

UU

02

12

1

02

12

1

U

Approximation of functionsApproximation of functions

Example: Find the best least square

approximation to on by a linear function .

Sol: (i) Clearly ,

but is not orthonormal

(ii) seek a function of the form

By calculation

is an orthonormal set of

(iii)

Thus the projection .

is the best linear least square approximation to on

xe 1,0

1

0

22

2)(20

20

,

)(min)(

1,0)(.,.

2

dxffffwhere

xPexPe

PxPFindeix

PxP

x

1,0,1 2Pxspan x,1

1)( axax

1,0xe

xeexee

ucucxP

)3(6)104())2

1(12)(3(31)1(

)( 2211

edxeueu

eeeu

xx

xx

33,

1,

1

0 22

1

01

2

10

2

1)(,1

1

0 aadxaxax

12

1

2

1x

1,02P

)

2

1(12,1 21 xuu

Approximation of trigonometric polynomialsApproximation of trigonometric polynomials

FACT: forms an

orthonormal set in with respect

to the inner product

Problem: Given a periodic function ,

find a trigonometric polynomial of degree n

which is a best least square approximation to .

Nnnxconx sin,,

2

1

,0 C

dxxgxfgf )()(

1,

2 )(xf

n

KKKn KxbKxa

axt

1

0 sincos2

)(

)(xf

Sol: It suffices to find the projection

of onto the subspace

The best approximation of

has coefficients

)(xf

nKKxconKxspan ,,1sin,,

2

1

tn

dxKxxfKxfb

dxKxxfKxfa

dxxffa

K

K

sin)(1

sin,

cos)(1

cos,

)(2

1

2

1,0

Example: Consider with

inner product of

(i) Check are orthonormal

(ii) Let

Similarly

(iii)

(iv)

,0 C

dxxgxfgf )()(

2

1,

nKeiKx ,,1,0

)(2

1

)(2

1

KK

iKxK

n

nK

iKxKn

iba

dxexfC

eCt

KK CC

KxbKxa

eCeC

KK

iKxK

iKxK

sincos

n

KKK

n

nK

iKxKn

KxbKxaa

eCt

1

0 sincos2

Cram-Schmidt Orthogonalization Cram-Schmidt Orthogonalization ProcessProcess

Question: Given a set of linear independent

vectors,

how to transform them into

orthogonal ones while

preserve spanning set.?

Given

,Clearly

Clearly

Similarly,

Clearly

We have the next result

Kxx 1

1

11 x

xu

}{}{ 11 xspanuspan

12

1221121 ,,

Px

PxuuuxP

},{},{& 212121 uuspanxxspanuu

23

233

2231132 ,,

Px

Pxuand

uuxuuxP

},,{},,{&, 3213212313 uuuspanxxxspanuuuu

1u 1P

2x

Theorem5.6.1: (The Cram-Schmidt process)

H. (i) be a basis for

an inner product space

(ii)

C. is an orthonormal basis nuu 1

nxx 1

V

j

K

jjKK

KK

KKK

uuxPwhere

nKPx

Pxu

xxu

11

1

11

1

11

,

1,,1,

,

Example: Find an orthonormal basis for

with inner product given by

where

Sol: Starting with a basis

3P

),()(,3

1i

ii xgxPgP

.1&0,1 321 xxx 2,,1 xx

32

32

03

2

22,

3

1

3

1,

2

03

1

3

1,

3

1

1

1

2

22

22

3

222

1

12

1

1

x

Px

Pxu

xxxxP

xPx

Pxu

xP

u

QR-DecompositionQR-DecompositionGiven

Let _______(1)

_________________(2)

_______________________(3)

Define

Where has orthonormal columns and is upper-triangular

naa 1

1111111 graar

1121121 , grggaP

1222 Par

2221122221

)3)(2(

222

122 grgrgrPa

r

Pag

i

K

iKii

K

iKiK grgagP

1

11,

1

11 ,

1

KKKK Par

KKK

K

iKiK

KK

KKK grrQ

r

Pag

1

11,

1

nnijnmn rRggQ

,1

QRAQ R

To solve with

Then, the example can be solved

By backsubstitution without finding

Inverse (if is square)

bxA nArankA nm )(&

bQxR

bxQR

A

Example: Solve

By direct calculation,

The solution can be obtained from

bA

x

x

x

2

1

1

1

0

2

1

1

0

4

0

2

4

2

2

1

3

2

1

R

Q

QRA

200

140

125

1

2

2

4

2

4

1

2

4

2

2

1

5

1

2

1

1

bQ

2

1

1

200

140

125