Lecture 11Fundamental Theorems of Linear

Algebra

Orthogonalily and Projection

Shang-Hua Teng

The Whole Picture• Rank(A) = m = n Ax=b has unique solution IR

FIR

0I

R

00FI

R

• Rank(A) = m < n Ax=b has n-m dimensional solution

• Rank(A) = n < m Ax=b has 0 or 1 solution

• Rank(A) < n, Rank(A) < m Ax=b has 0 or n-rank(A) dimensions

Basis and Dimension of a Vector Space• A basis for a vector space is a sequence of

vectors that – The vectors are linearly independent– The vectors span the space: every vector in the

vector can be expressed as a linear combination of these vectors

Basis for 2D and n-D

• (1,0), (0,1)• (1 1), (-1 –2)

• The vectors v1,v2,…vn are basis for Rn if and only if they are columns of an n by n invertible matrix

Column and Row Subspace• C(A): the space spanned by columns of A

– Subspace in m dimensions– The pivot columns of A are a basis for its column space

• Row space: the space spanned by rows of A– Subspace in n dimensions– The row space of A is the same as the column space of AT, C(AT)– The pivot rows of A are a basis for its row space– The pivot rows of its Echolon matrix R are a basis for its row

space

Important Property I: Uniqueness of Combination

• The vectors v1,v2,…vn are basis for a vector space V, then for every vector v in V, there is a unique way to write v as a combination of v1,v2,…vn .

v = a1 v1+ a2 v2+…+ an vn

v = b1 v1+ b2 v2+…+ bn vn

• So: 0=(a1 - b1) v1 + (a2 -b2 )v2+…+ (an -bn )vn

Important Property II: Dimension and Size of Basis

• If a vector space V has two set of bases– v1,v2,…vm . V = [v1,v2,…vm ]– w1,w2,…wn . W= [w1,w2,…wn ].

• then m = n– Proof: assume n > m, write W = VA– A is m by n, so Ax = 0 has a non-zero solution– So VAx = 0 and Wx = 0

• The dimension of a vector space is the number of vectors in every basis– Dimension of a vector space is well defined

Dimensions of the Four SubspacesFundamental Theorem of Linear

Algebra, Part I• Row space: C(AT) – dimension = rank(A)• Column space: C(A)– dimension = rank(A)• Nullspace: N(A) – dimension = n-rank(A)• Left Nullspace: N(AT) – dimension = m –rank(A)•

Orthogonality and Orthogonal Subspaces

• Two vectors v and w are orthogonal if 0 vwwvwv TT

• Two vector subspaces V and W are orthogonal if

0 , and allfor wvWwVv T

Example: Orthogonal Subspace in 5 Dimensions

11000

,

10000

00110

,

00011

,

00001

CC

The union of these two subspaces is R5

Orthogonal Complement

• Suppose V is a vector subspace a vector space W

• The orthogonal complement of V is}such that { VwWwV

• Orthogonal complement is itself a vector subspace

)dim()dim()dim( WVV

Dimensions of the Four SubspacesFundamental Theorem of Linear

Algebra, Part I• Row space: C(AT) – dimension = rank(A)• Column space: C(A)– dimension = rank(A)• Nullspace: N(A) – dimension = n-rank(A)• Left Nullspace: N(AT) – dimension = m –rank(A)•

Orthogonality of the Four SubspacesFundamental Theorem of Linear

Algebra, Part II• The nullspace is the

orthogonal complement of the row space in Rn

• The left Nullspace is the orthogonal complement of the column space in Rm

•

)()( TACAN

))(()( ACAN T

Proof• The nullspace is the

orthogonal complement of the row space in Rn

)()( TACAN

0

implying:)(

0:)(

AxyAxyxyAyAx

RyyAAC

AxxAN

TTTTTT

mTT

The Whole Picture

C(AT)

N(A)

Rn

Rm

C(A)

N(AT)

xn A xn= 0

xr

b

A xr= b

nr xxx

A x= b

dim r dim r

dim n- r

dim m- r

Uniqueness of The Typical Solution

• Every vector in the column space comes from one and only one vector xr from the row space

• Proof: suppose there are two xr , yr from the row space such that Axr =A yr =b, then

Axr -A yr = A(xr -yr ) = 0

(xr -yr ) is in row space and nullspace hence must be 0• The matching of dim in row and column spaces

Deep Secret of Linear AlgebraPseudo-inverse

• Throw away the two null spaces, there is an r by r invertible matrix hiding insider A.

• In some sense, from the row space to the column space, A is invertible

• It maps an r-space in n space to an r-space in m-space

Invertible Matrices

• Any n linearly independent vector in Rn must span Rn . They are basis.

• So Ax = b is always uniquely solvable

• A is invertible

Projection

• Projection onto an axis

(a,b)

x axis is a vector subspace

Projection onto an Arbitrary Line Passing through 0

(a,b)

Projection on to a Plane

Projection onto a Subspace

• Input: 1. Given a vector subspace V in Rm

2. A vector b in Rm…• Desirable Output:

– A vector in x in V that is closest to b– The projection x of b in V– A vector x in V such that (b-x) is orthogonal

to V

How to Describe a Vector Subspace V in Rm

• If dim(V) = n, then V has n basis vectors– a1, a2, …, an

– They are independent

• V = C(A) where A = [a1, a2, …, an]

Projection onto a Subspace

• Input: 1. Given n independent vectors a1, a2, …, an in Rm

2. A vector b in Rm…• Desirable Output:

– A vector in x in C([a1, a2, …, an]) that is closest to b

– The projection x of b in C([a1, a2, …, an])

– A vector x in V such that (b-x) is orthogonal to C([a1, a2, …, an])

Think about this Picture

C(AT)

N(A)

Rn

Rm

C(A)

N(AT)

xn A xn= 0

xr

b

A xr= b

nr xxx

A x= b

dim r dim r

dim n- r

dim m- r