mathcamp2013 fa slides

8/12/2019 Mathcamp2013 Fa Slides

1/53

OutlineVector Spaces

Hilbert SpacesFunctionals and Operators (Matrices)

Linear Operators

Functional Analysis Review

Lorenzo Rosascoslides courtesy of Andre Wibisono

9.520: Statistical Learning Theory and Applications

September 9, 2013

L. Rosasco Functional Analysis Review
http://find/http://goback/


2/53



Linear Operators

1 Vector Spaces

2

Hilbert Spaces

3 Functionals and Operators (Matrices)

4 Linear Operators

http://find/


3/53



Linear Operators

Vector Space

A vector space is a set Vwith binary operations

+ : V VV and : R VV

such that for all a, b R and v,w, xV:1 v +w= w +v2 (v +w) + x= v + (w + x)3 There exists 0 Vsuch that v +0=v for all v V4 For every v Vthere exists v Vsuch that v + (v) =0

5 a(bv) = (ab)v6 1v=v7 (a + b)v= av + bv8 a(v +w) =av + aw

http://find/


4/53



Linear Operators

Vector Space

A vector space is a set Vwith binary operations

+ : V VV and : R VV

such that for all a, b R and v,w, xV:1 v +w= w +v2 (v +w) + x= v + (w + x)3 There exists 0 Vsuch that v +0=v for all v V4 For every v Vthere exists v Vsuch that v + (v) =0

5 a(bv) = (ab)v6 1v=v7 (a + b)v= av + bv8 a(v +w) =av + aw

Example: Rn, space of polynomials, space of functions.


O li


5/53



Linear Operators

Inner Product

An inner product is a function , : V V R suchthat for all a, b R and v,w, xV:


O tli
http://find/


6/53



Linear Operators

Inner Product


1 v,w=w,v

2 av + bw, x=av, x + bw, x

3 v,v 0 and v,v=0 if and only ifv=0.


Outline
http://find/


7/53



Linear Operators

Inner Product


1 v,w=w,v



v,w Vare orthogonal ifv,w=0.


Outline
http://find/


8/53



Linear Operators

Inner Product


1 v,w=w,v




Given WV, we have V=W W, whereW ={vV |v,w=0 for all w W}.


Outline
http://find/


9/53



Linear Operators

Inner Product


1 v,w=w,v




Given WV, we have V=W W, whereW ={vV |v,w=0 for all w W}.

Cauchy-Schwarz inequality: v,w v,v1/2w,w1/2.


Outline
http://find/


10/53



Linear Operators

Norm

Can define norm from inner product: v=v,v1/2.


Outline
http://find/


11/53

Vector SpacesHilbert Spaces

Functionals and Operators (Matrices)Linear Operators

Norm

A norm is a function : V R such that for all a Rand v,w V:

1 v 0, and v=0 if and only ifv=0

2 av= |a| v

3 v +w v + w

Can define norm from inner product: v=v,v1/2.


Outline
http://find/


12/53



Metric

Can define metric from norm: d(v,w) =v w.


Outline
http://find/


13/53



Metric

A metric is a function d : V V R such that for allv,w, x V:

1 d(v,w) 0, and d(v,w) =0 if and only ifv=w

2 d(v,w) =d(w,v)

3 d(v,w) d(v, x) + d(x,w)

Can define metric from norm: d(v,w) =v w.


OutlineV S
http://find/


14/53



Basis

B={v1, . . . ,vn} is a basis ofV if every v Vcan beuniquely decomposed as

v=a1v1+ + anvn

for some a1, . . . , an R.


OutlineV t S
http://find/


15/53



Basis

B={v1, . . . ,vn} is a basis ofV if every v Vcan beuniquely decomposed as

v=a1v1+ + anvn

for some a1, . . . , an R.

An orthonormal basis is a basis that is orthogonal

(vi,vj=0 for i=j) and normalized (vi=1).


http://find/


16/53



1 Vector Spaces

2 Hilbert Spaces


4 Linear Operators


http://find/


17/53



Hilbert Space, overview

Goal: to understand Hilbert spaces (complete innerproduct spaces) and to make sense of the expression

f=

i=1

f, ii, f H

Need to talk about:

1 Cauchy sequence

2 Completeness

3 Density

4 Separability


http://find/


18/53



Cauchy Sequence

Recall: limn

xn =x if for every >0 there exists

N N such that x xn< whenever n N.


http://find/


19/53



Cauchy Sequence

Recall: limn



(xn)nN is a Cauchy sequence if for every >0 thereexists N Nsuch that xm xn< whenever m, n N.


http://find/


20/53

pHilbert Spaces


Cauchy Sequence

Recall: limn



(xn)nN is a Cauchy sequence if for every >0 thereexists N Nsuch that xm xn< whenever m, n N.

Every convergent sequence is a Cauchy sequence (why?)


http://find/


21/53

pHilbert Spaces


Completeness

A normed vector space V is complete if every Cauchysequence converges.


http://find/


22/53


Linear Operators

Completeness

A normed vector space V is complete if every Cauchysequence converges.

Examples:1 Q is not complete.

2 R is complete (axiom).

3 Rn is complete.

4 Every finite dimensional normed vector space (over R) iscomplete.



Hilb S
http://find/


23/53


Linear Operators

Hilbert Space

A Hilbert space is a complete inner product space.



Hilb t S
http://find/


24/53


Linear Operators

Hilbert Space

A Hilbert space is a complete inner product space.

Examples:

1 Rn

2 Every finite dimensional inner product space.

3 2 ={(an)n=1 |an R,

n=1 a

2n


25/53


Hilbert Spaces


26/53


Linear Operators

Density

Y is dense in X ifY=X.

Examples:

1 Q is dense in R.

2 Qn is dense in Rn.

3 Weierstrass approximation theorem: polynomials are densein continuous functions (with the supremum norm, on

compact domains).



Hilbert Spaces
http://find/


27/53


Linear Operators

Separability

X is separable if it has a countable dense subset.



Hilbert Spaces
http://find/


28/53


Linear Operators

Separability

X is separable if it has a countable dense subset.

Examples:1 R is separable.

2 Rn is separable.

3 2, L2([0, 1]) are separable.



Hilbert Spaces
http://find/


29/53

pFunctionals and Operators (Matrices)

Linear Operators

Orthonormal Basis

A Hilbert space has a countable orthonormal basis if andonly if it is separable.

Can write:

f=

i=1

f, ii for all fH.



Hilbert Spaces
http://find/


30/53

pFunctionals and Operators (Matrices)

Linear Operators

Orthonormal Basis

A Hilbert space has a countable orthonormal basis if andonly if it is separable.

Can write:

f=

i=1

f, ii for all fH.

Examples:1 Basis of2 is (1 , 0 , . . . , ), (0 , 1 , 0 , . . . ), (0 , 0 , 1 , 0 , . . . ), . . .

2 Basis ofL2([0, 1]) is 1, 2 sin 2nx,2cos2nx for n N


OutlineVector SpacesHilbert Spaces
http://find/


31/53


1 Vector Spaces

2 Hilbert Spaces


4 Linear Operators



F i l d O (M i )
http://find/


32/53


Maps

Next we are going to review basic properties of maps on a

Hilbert space.functionals: :H R

linear operators A:HH, such thatA(af + bg) =aAf + bAg, with a, b R and f, gH.



F ti l d O t (M t i )
http://find/


33/53


Representation of Continuous Functionals

Let H be a Hilbert space and gH, then

g(f) =f, g , fH

is a continuous linear functional.Riesz representation theorem

The theorem states that every continuous linear functional can be written uniquely in the form,

(f) =f, g

for some appropriate element g H.

http://find/


34/53


Functionals and Operators (Matrices)


35/53


Matrix

Every linear operator L : Rm Rn can be represented byan m n matrix A.

IfA Rmn, the transpose ofAis A Rnm satisfying

Ax,yRm = (Ax)y=xAy=x, AyRn

for every x Rn and y Rm.



http://find/


36/53


Matrix

Every linear operator L : Rm Rn can be represented byan m n matrix A.

IfA Rmn, the transpose ofAis A Rnm satisfying

Ax,yRm = (Ax)y=xAy=x, AyRn

for every x Rn and y Rm.

A is symmetric ifA =A.



http://find/


37/53


Eigenvalues and Eigenvectors

Let A Rnn. A nonzero vector v Rn is an eigenvectorofA with corresponding eigenvalue R ifAv=v.



38/53


39/53


40/53


Functionals and Operators (Matrices)Li O


41/53

Linear Operators

Singular Value Decomposition

Every A Rmn can be written as

A=UV,

where U Rmm is orthogonal, Rmn is diagonal,and V Rnn is orthogonal.



Functionals and Operators (Matrices)Li O t
http://find/


42/53

Linear Operators

Singular Value Decomposition

Every A Rmn can be written as

A=UV,

where U Rmm is orthogonal, Rm

n is diagonal,

and V Rnn is orthogonal.

Singular system:

Avi =iui AAui =

2

i

ui

Aui =ivi AAvi =

2ivi



Functionals and Operators (Matrices)Linea O e ato s
http://find/


43/53

Linear Operators

Matrix Norm

The spectral norm ofA Rmn is

Aspec =max(A) =

max(AA) =

max(AA).



http://find/


44/53

Linear Operators

Matrix Norm

The spectral norm ofA Rmn is

Aspec =max(A) =

max(AA) =

max(AA).

The Frobenius norm ofA Rmn is

AF =

m

i=1

n

j=1

a2ij =min{m,n}

i=1

2i .



http://find/


45/53

Linear Operators

Positive Definite Matrix

A real symmetric matrix A Rmm is positive definite if

xtAx >0, x Rm.

A positive definite matrix has positive eigenvalues.

Note: for positive semi-definite matrices > is replaced by .



http://find/


46/53

Linear Operators

1 Vector Spaces

2 Hilbert Spaces


4 Linear Operators



http://find/


47/53

Op

Linear Operator

An operator L : H1H2 is linear if it preserves the linearstructure.




Linear Operators
http://find/


48/53

p

Linear Operator


A linear operator L : H1H2 is bounded if there existsC >0 such that

LfH2 CfH1 for all fH1.




Linear Operators
http://goforward/http://find/http://goback/


49/53

Linear Operator


A linear operator L : H1H2 is bounded if there existsC >0 such that

LfH2 CfH1 for all fH1.

A linear operator is continuous if and only if it is bounded.




Linear Operators
http://find/


50/53

Adjoint and Compactness

The adjoint of a bounded linear operator L : H1H2 is abounded linear operator L : H2H1 satisfying

Lf, gH2 =f, LgH1 for all fH1, gH2.

L is self-adjoint ifL =L. Self-adjoint operators have realeigenvalues.




Linear Operators
http://find/


51/53

Adjoint and Compactness

The adjoint of a bounded linear operator L : H1H2 is abounded linear operator L : H2H1 satisfying

Lf, gH2 =f, LgH1 for all fH1, gH2.

L is self-adjoint ifL =L. Self-adjoint operators have realeigenvalues.

A bounded linear operator L : H1H2 is compact if the

image of the unit ball in H1 has compact closure in H2.




Linear Operators
http://find/


52/53

Spectral Theorem for Compact Self-Adjoint Operator

Let L : HH be a compact self-adjoint operator. Thenthere exists an orthonormal basis ofH consisting of theeigenfunctions ofL,

Li =ii

and the only possible limit point ofi as i is 0.




Linear Operators
http://find/


53/53

Spectral Theorem for Compact Self-Adjoint Operator

Let L : HH be a compact self-adjoint operator. Thenthere exists an orthonormal basis ofH consisting of theeigenfunctions ofL,

Li =ii

and the only possible limit point ofi as i is 0.Eigendecomposition:

L=i=1

ii, i.

http://find/

mathcamp2013 fa slides

Documents