linear algebra review by tim k. marks ucsd borrows heavily from: jana kosecka kosecka/cs682.html...

Linear Algebra Review

By Tim K. Marks

UCSD

Borrows heavily from:

Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html

Virginia de Sa (UCSD)Cogsci 108F Linear Algebra review

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Vectors

The length of x, a.k.a. the norm or 2-norm of x, is

e.g.,


are needed to see this picture.QuickTime™ and aTIFF (LZW) decompressorare needed to see this picture.



€

x = x12 + x2

2 +L + xn2

€

x = 32 + 22 + 52 = 38

Good Review Materials

http://www.imageprocessingbook.com/DIP2E/dip2e_downloads/review_material_downloads.htm

(Gonzales & Woods review materials)

Chapt. 1: Linear Algebra Review

Chapt. 2: Probability, Random Variables, Random Vectors

http://www.imageprocessingbook.com/DIP2E/dip2e_downloads/review_material_downloads.htm





Online vector addition demo:

http://www.pa.uky.edu/~phy211/VecArith/index.html

http://www.pa.uky.edu/~phy211/VecArith/index.html

Vector Addition

vvuu

u+vu+v

Vector Subtraction

uuvv

u-vu-v



Example (on board)

Inner product (dot product) of two vectors

€

a =

6

2

−3

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

€

b =

4

1

5

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

€

a ⋅b = aTb

€

= 6 2 −3[ ]

4

1

5

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

€

=6 ⋅4 + 2 ⋅1+ (−3) ⋅5

€

=11

Inner (dot) Product

vv

uu

The inner product is a The inner product is a SCALAR.SCALAR.





€

uTv = 0 ⇔ u ⊥v

Transpose of a Matrix

Transpose:Transpose:

Example of symmetric matrixExample of symmetric matrix

€

Cm×n = ATn×m

€

c ij = a ji

€

(A + B)T = AT + BT

€

(AB)T = BT AT

Examples:Examples:

€

6 2

1 5

⎡

⎣ ⎢

⎤

⎦ ⎥

T

=6 1

2 5

⎡

⎣ ⎢

⎤

⎦ ⎥

€

6 2

1 5

3 8

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

T

=6 1 3

2 5 8

⎡

⎣ ⎢

⎤

⎦ ⎥

If If , we say , we say AA is is symmetricsymmetric..

€

AT = A

€

6 2

1 5

⎡

⎣ ⎢

⎤

⎦ ⎥⋅

2 5

3 1

⎡

⎣ ⎢

⎤

⎦ ⎥=

18 32

17 10

⎡

⎣ ⎢

⎤

⎦ ⎥

Matrix ProductProduct:Product:

Examples:Examples:

A and B must have A and B must have compatible dimensionscompatible dimensions

In Matlab: >> A*B€

Cn× p = An×mBm× p

€

c ij = aikbkj

k=1

m

∑

Matrix Multiplication is not commutative:Matrix Multiplication is not commutative:

€

An×nBn×n ≠ Bn×n An×n€

2 5

3 1

⎡

⎣ ⎢

⎤

⎦ ⎥⋅

6 2

1 5

⎡

⎣ ⎢

⎤

⎦ ⎥=

17 29

19 11

⎡

⎣ ⎢

⎤

⎦ ⎥

€

c ij = aij + bij

€

Cn×m = An×m + Bn×m

Matrix Sum

Sum:Sum:

A and B must have the A and B must have the same dimensionssame dimensions

Example:Example:

€

2 5

3 1

⎡

⎣ ⎢

⎤

⎦ ⎥+

6 2

1 5

⎡

⎣ ⎢

⎤

⎦ ⎥=

8 7

4 6

⎡

⎣ ⎢

⎤

⎦ ⎥

€

deta11 a12

a21 a22

⎡

⎣ ⎢

⎤

⎦ ⎥=

a11 a12

a21 a22

= a11a22 − a21a12

€

det

a11 a12 a13

a21 a22 a23

a31 a32 a33

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥= a11

a22 a23

a32 a33

− a12

a21 a23

a31 a33

+ a13

a21 a22

a31 a32

Determinant of a Matrix

Determinant:Determinant:

€

det2 5

3 1

⎡

⎣ ⎢

⎤

⎦ ⎥= 2 −15 = −13Example:Example:

A must be squareA must be square

Determinant in Matlab



Inverse of a Matrix

If A is a square matrix, the inverse of A, called A-1, satisfies

AA-1 = I and A-1A = I,

Where I, the identity matrix, is a diagonal matrix with all 1’s on the diagonal.

€

I2 =1 0

0 1

⎡

⎣ ⎢

⎤

⎦ ⎥

€

I3 =

1 0 0

0 1 0

0 0 1

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

Inverse of a 2D Matrix

€

6 2

1 5

⎡

⎣ ⎢

⎤

⎦ ⎥

−1

⋅6 2

1 5

⎡

⎣ ⎢

⎤

⎦ ⎥=

1

28

5 −2

−1 6

⎡

⎣ ⎢

⎤

⎦ ⎥⋅

6 2

1 5

⎡

⎣ ⎢

⎤

⎦ ⎥=

1

28

28 0

0 28

⎡

⎣ ⎢

⎤

⎦ ⎥=

1 0

0 1

⎡

⎣ ⎢

⎤

⎦ ⎥

€

6 2

1 5

⎡

⎣ ⎢

⎤

⎦ ⎥

−1

=1

28

5 −2

−1 6

⎡

⎣ ⎢

⎤

⎦ ⎥Example:Example:





Inverses in Matlab



Trace of a matrix



Example (on board)

Matrix Transformation: Scale

A square, diagonal matrix scales each dimension by the corresponding diagonal element.

Example:

€

2 0 0

0 .5 0

0 0 3

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

6

8

10

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ =

12

4

30

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥



Linear Independence• A set of vectors is linearly dependent if one of

the vectors can be expressed as a linear combination of the other vectors.

Example:

€

1

0

0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ,

0

1

0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ,

2

1

0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

• A set of vectors is linearly independent if none of the vectors can be expressed as a linear combination of the other vectors.

Example:

€

1

0

0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ,

0

1

0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ,

2

1

3

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

Rank of a matrix• The rank of a matrix is the number of linearly

independent columns of the matrix.Examples:

has rank 2

€

1 0 2

0 1 1

0 0 0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

• Note: the rank of a matrix is also the number of linearly independent rows of the matrix.

has rank 3

€

1 0 2

0 1 0

0 0 1

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

Singular Matrix

All of the following conditions are equivalent. We say a square (n n) matrix is singular if any one of these conditions (and hence all of them) is satisfied.– The columns are linearly dependent

– The rows are linearly dependent

– The determinant = 0

– The matrix is not invertible

– The matrix is not full rank (i.e., rank < n)

Linear Spaces

A linear space is the set of all vectors that can be expressed as a linear combination of a set of basis vectors. We say this space is the span of the basis vectors.– Example: R3, 3-dimensional Euclidean space, is

spanned by each of the following two bases:

€

1

0

0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ,

0

1

0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ,

0

0

1

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

€

1

0

0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ,

0

1

2

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ,

0

0

1

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

Linear Subspaces

A linear subspace is the space spanned by a subset of the vectors in a linear space.– The space spanned by the following vectors is a

two-dimensional subspace of R3.

€

1

0

0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ,

0

1

0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

€

1

1

0

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ,

0

0

1

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

What does it look like?

What does it look like?

– The space spanned by the following vectors is a two-dimensional subspace of R3.

Orthogonal and Orthonormal Bases

n linearly independent real vectors span Rn, n-dimensional Euclidean space

• They form a basis for the space.

– An orthogonal basis, a1, …, an satisfies

ai aj = 0 if i ≠ j

– An orthonormal basis, a1, …, an satisfies

ai aj = 0 if i ≠ j

ai aj = 1 if i = j

– Examples.

Orthonormal Matrices

A square matrix is orthonormal (also called unitary) if its columns are orthonormal vectors.– A matrix A is orthonormal iff AAT = I.

• If A is orthonormal, A-1 = AT

AAT = ATA = I.

– A rotation matrix is an orthonormal matrix with determinant = 1.

• It is also possible for an orthonormal matrix to have determinant = –1. This is a rotation plus a flip (reflection).



Matrix Transformation: Rotation by an Angle











2D rotation matrix:



http://www.math.ubc.ca/~cass/courses/m309-8a/java/m309gfx/eigen.html

http://www.math.ubc.ca/~cass/courses/m309-8a/java/m309gfx/eigen.html

Some Properties of Eigenvalues and Eigenvectors

– If 1, …, n are distinct eigenvalues of a matrix, then the corresponding eigenvectors e1, …, en are linearly independent.

– If e1 is an eigenvector of a matrix with corresponding

eigenvalue 1, then any nonzero scalar multiple of e1 is

also an eigenvector with eigenvalue 1.

– A real, symmetric square matrix has real eigenvalues, with orthogonal eigenvectors (can be chosen to be orthonormal).

SVD: Singular Value DecompositionAny matrix A (m n) can be written as the product of three

matrices:A = UDV T

where

– U is an m m orthonormal matrix

– D is an m n diagonal matrix. Its diagonal elements, 1, 2, …, are

called the singular values of A, and satisfy 1 ≥ 2 ≥ … ≥ 0.

– V is an n n orthonormal matrix

Example: if m > n

€

• • •• • •• • •• • •• • •

⎡

⎣

⎢ ⎢ ⎢ ⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ⎥ ⎥ ⎥

=

↑ ↑ ↑ ↑

| | | |

u1 u2 u3 L um

| | | |

↓ ↓ ↓ ↓

⎡

⎣

⎢ ⎢ ⎢ ⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ⎥ ⎥ ⎥

σ 1 0 0

0 σ 2 0

0 0 σ n

0 0 0

0 0 0

⎡

⎣

⎢ ⎢ ⎢ ⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ⎥ ⎥ ⎥

← v1T →

M M M

← vnT →

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

A U D V T

SVD in Matlab>> a = [1 2 3; 2 7 4; -3 0 6; 2 4 9; 5 -8 0]a = 1 2 3 2 7 4 -3 0 6 2 4 9 5 -8 0

>> [u,d,v] = svd(a)u = -0.24538 0.11781 -0.11291 -0.47421 -0.82963 -0.53253 -0.11684 -0.52806 -0.45036 0.4702 -0.30668 0.24939 0.79767 -0.38766 0.23915 -0.64223 0.44212 -0.057905 0.61667 -0.091874 0.38691 0.84546 -0.26226 -0.20428 0.15809

d =

14.412 0 0

0 8.8258 0

0 0 5.6928

0 0 0

0 0 0

v =

0.01802 0.48126 -0.87639

-0.68573 -0.63195 -0.36112

-0.72763 0.60748 0.31863

Some Properties of SVD

– The rank of matrix A is equal to the number of nonzero singular values i

– A square (n n) matrix A is singular if and only if

at least one of its singular values 1, …, n is zero.

Geometric Interpretation of SVDIf A is a square (n n) matrix,

– U is a unitary matrix: rotation (possibly plus flip)– D is a scale matrix– V (and thus V T) is a unitary matrix

Punchline: An arbitrary n-D linear transformation is equivalent to a rotation (plus perhaps a flip), followed by a scale transformation, followed by a rotation

Advanced: y = Ax = UDV Tx– V T expresses x in terms of the basis V.– D rescales each coordinate (each dimension)– The new coordinates are the coordinates of y in terms of the basis U

€

• • •• • •• • •

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ =

↑ L ↑

u1 L un

↓ L ↓

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

σ 1 0 0

0 σ 2 0

0 0 σ n

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

← v1T →

M M M

← vnT →

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

A U D V T

linear algebra review by tim k. marks ucsd borrows heavily from: jana kosecka kosecka/cs682.html...

Documents