svd review

CS 395/495-26: Spring 2004CS 395/495-26: Spring 2004

IBMR: IBMR: Singular Value Singular Value DecompositionDecomposition(SVD Review)(SVD Review)

Jack TumblinJack Tumblin

[email protected]@cs.northwestern.edu

Matrix Multiply: A Change-of-AxesMatrix Multiply: A Change-of-Axes

• Matrix Multiply: Ax = bMatrix Multiply: Ax = b– x and b are column vectorsx and b are column vectors– A has m rows, n columnsA has m rows, n columns

xx11

……xxnn

aa11 11 … a … a1n1n

…… … … aam1m1 … … a amnmn

= = bb11

……bbmm

xx11

xx22

bb33

bb22

bb11Ax=bAx=bxx bb

Input (N dim) Input (N dim) (2D)(2D) Output (M-dim) Output (M-dim) (3D)(3D)

Matrix Multiply: A Change-of-AxesMatrix Multiply: A Change-of-Axes

• Matrix Multiply: Ax = bMatrix Multiply: Ax = b– x and b are column vectorsx and b are column vectors– A has m rows, n columnsA has m rows, n columns

• Matrix multiply is just a set of dot-products;Matrix multiply is just a set of dot-products;it ‘changes coordinates’ for vector x, makes b.it ‘changes coordinates’ for vector x, makes b.– Rows of A = ARows of A = A11,A,A22,A,A33,... = new coordinate axes,... = new coordinate axes

– Ax = a dot product for each new axisAx = a dot product for each new axis

xx11

……xxnn

aa11 11 … a … a1n1n

…… … … aam1m1 … … a amnmn

= = bb11

……bbmm

xx11

xx22

AA22

AA11 AA33

bb33

bb22

bb11Ax=bAx=bxx bb

Input (N dim) Input (N dim) (2D)(2D) Output (M-dim) Output (M-dim) (3D)(3D)

• Sphere of all unit-length Sphere of all unit-length xx ellipsoid of ellipsoid of bb• Output ellipsoid’s axes are Output ellipsoid’s axes are

always perpendicular (true for any ellipsoid)always perpendicular (true for any ellipsoid)

**THUS** we can make **THUS** we can make ┴┴, unit-length axes…, unit-length axes…

How Does Matrix ‘stretch space’?How Does Matrix ‘stretch space’?

xx11

xx22

bb33

bb22

bb11Ax=bAx=b11

Input (N dim.)Input (N dim.) Output (M dim.)Output (M dim.)

uu11uu22

xx11

ellipsoid (disk)ellipsoid (disk)

bb

• Sphere of all unit-length Sphere of all unit-length xx ellipsoid of ellipsoid of bb• Output ellipsoid’s axes form Output ellipsoid’s axes form

orthonormal basis vectors orthonormal basis vectors UUii::• Basis vectors Basis vectors UUi i are columns of U matrixare columns of U matrix

SVD(A) = SVD(A) = UUSVSVTT columns of U = columns of U = OUTPUTOUTPUT basis vectors basis vectors

SVD finds: ‘Output’ Vectors SVD finds: ‘Output’ Vectors UU, and…, and…

xx11

xx22

bb33

bb22

bb11Ax=bAx=b11

Input (N dim.)Input (N dim.) Output (M dim.)Output (M dim.)

uu11uu22

xx11

ellipsoid (disk)ellipsoid (disk)

bb

SVD finds: …‘Input’ Vectors SVD finds: …‘Input’ Vectors VV, and…, and…

• For each For each UUii , make a matching , make a matching VVii that: that:

– Transforms to Transforms to UUii (with scaling (with scaling ssii) : ) : ssii (A V (A Vii) = U) = Uii

– Forms an orthonormal basis of input spaceForms an orthonormal basis of input space

SVD(A) = USSVD(A) = USVVTT columns of V = columns of V = INPUTINPUT basis vectors basis vectors

xx11

xx22

Ax=bAx=bxx

Input (N dim.)Input (N dim.)

bb33

bb22

bb11

Output (M dim.)Output (M dim.)

uu11uu22

vv22

vv11bb

SVD finds: …‘Scale’ factors SVD finds: …‘Scale’ factors SS..

• We have Unit-length Output We have Unit-length Output UUii, vectors,, vectors,

• Each from a Unit-length Input Each from a Unit-length Input VVii vector, so vector, so

• So we need ‘scale factor’ So we need ‘scale factor’ (singular values)(singular values) ssi i

to link input to output: to link input to output: ssii (A V (A Vii) = U) = Uii

SVD(A) = USVD(A) = USSVVTT

xx11

xx22

Ax=bAx=bxx


bb33

bb22

bb11


uu11uu22

vv22

vv11

ss22

ss11

bb

SVD Review: What is it?SVD Review: What is it?

• Finish: Finish: SVD(A) = SVD(A) = USVUSVTT

– add ‘add ‘missingmissing’ ’ UUii or or VVii, define these , define these ssii=0=0..

– Singular matrix Singular matrix SS: diagonals are : diagonals are ssii : ‘v-to-u scale factors’ : ‘v-to-u scale factors’

– Matrix Matrix UU, Matrix , Matrix VV have columns have columns UUii and and VVii

xx11

xx22

Ax=bAx=bxx


bb33

bb22

bb11


uu11

uu33

uu22

vv22

vv11

A = A = USVUSVT T ‘Find Input & Output Axes, Linked by Scale’‘Find Input & Output Axes, Linked by Scale’

bb

‘‘Let SVDs Explain it All for You’Let SVDs Explain it All for You’– cool! U and V are Orthonormal! cool! U and V are Orthonormal! UU-1-1 = U = UTT, V, V-1-1 = V = VTT

– Rank(A)?Rank(A)? == # of non-zero singular values == # of non-zero singular values ssii – ‘‘ill conditioned’?ill conditioned’? Some Some ssii are nearly zero are nearly zero– ‘‘Invert’ a non-square matrix A ?Invert’ a non-square matrix A ? Amazing! Amazing!

pseudo-inverse:pseudo-inverse: A=USVA=USVTT; A VS; A VS-1-1UUTT = I; = I; so so AA-1 -1 == VV SS-1 -1 UUTT

xx11

xx22

Ax=bAx=bxx


bb33

bb22

bb11


uu11

uu33

uu22

vv22

vv11


‘‘Solve for the Null Space’ ASolve for the Null Space’ Axx=0=0

– Easy! x is the Easy! x is the VVii axis axis that doesn’t affect the output (that doesn’t affect the output (ssii = 0 = 0))– are are allall ssii nonzero? then the only answer is nonzero? then the only answer is x=0.x=0.– Null ‘space’Null ‘space’ because V because Vii is a DIMENSION--- x = Vi * a is a DIMENSION--- x = Vi * a– More than 1 zero-valued sMore than 1 zero-valued sii? null space >1 dimension? null space >1 dimension

... ... x = ax = aVVi1i1 + b + bVVi2i2 + … + …

xx11

xx22

Ax=bAx=bxx


bb33

bb22

bb11


uu11

uu33

uu22

vv22

vv11


More on SVDs: HandoutMore on SVDs: Handout

• Zisserman Appendix 3, pg 556 (cryptic)Zisserman Appendix 3, pg 556 (cryptic)

• Linear Algebra booksLinear Algebra books• Good online introduction: (your handout)Good online introduction: (your handout)

Leach, S. “Singular Value Decomposition – A Leach, S. “Singular Value Decomposition – A primer,” Unpublished Manuscript, Department of primer,” Unpublished Manuscript, Department of Computer Science, Brown University. Computer Science, Brown University. http://www.cs.brown.edu/research/ai/dynamics/tutorial/Postscript/http://www.cs.brown.edu/research/ai/dynamics/tutorial/Postscript/http://nextlib.lifegoo.com/user/jack/article/1791http://nextlib.lifegoo.com/user/jack/article/1791

• Google on Singular Value Decomposition...Google on Singular Value Decomposition...

ENDEND

svd review

Technology