Vector SpacesVector Spaces

Axioms of a Vector SpaceAxioms of a Vector SpaceDefinition : Let V be a non empty set of vectors with 2 operations :i. Vector addition : u, v є V u + v є Vii S l lti li ti V k V h k i l Th V

closureii. Scalar multiplication: u є V k u є V where k is scalar. Then, V

is called a Vector Space if the following 8 axioms hold for any u, v, w є V

c osu e

)2)()()1

uu00uwvuwvu

vectorzeroeassociativ

)4inverse additive )()3

uvvu0uuuu

ecommutativ

)()()7,)()6

)()5uuuvuvu

bbscalarsbababascalariskkkk

1)8)()()7

uuuu

baab

too many conditions to check !!

SubspacesSubspacesDefinition : Let V be a vector space and let W be a subset of V. Then, W

is a subspace of V if W itself is a vector space

Theorem : Suppose W is a subset of V. Then, W is a subspace if

a) The zero vector 0 belongs to W

b) For every u, v є W (au + bv) є W

We don’t need to verify that the 8 axioms of a vector space hold !

Remark : Two trivial subspaces of V are { 0 } and V itself

c}ba:c)b{(aUi eentriesequal

with Rin vectorsall ofconsist Let U .R VConsider : Example 33

Ub)(1,1,1) a( v u

b)b,(b,a)a, (a, u

(ii) U 0 (i)

c}ba:c)b,{(a, Ui.e. entries, equal

vsubspace a is U

Ub)b,(b, v

“Abstract” Subspaces ExamplesAbstract Subspaces ExamplesExamples : (prove by showing that the 2 conditions of a subspace hold)

Pn(t): polynomials of degree less than or equal to n. Note that it is no longer b if li it th d f th l i l t h ?a subspace if we limit the degree of the polynomial to n ; why ?

Rn : n tuple (vector) of real numbers (a a a )Rn : n-tuple (vector) of real numbers (a1, a2, ……, an)

M : m x n real matricesMm,n : m x n real matrices

F : all real functions f(x). What about continuous or integrable or ( ) gdifferentiable functions ?

More Subspaces ExamplesMore Subspaces ExamplesExample : Let V = Mn,n. Let be a subset of all upper (or lower) triangular

t i1W

matrices

W1 is a subspace

Let W be the subset of all symmetric matrices (all zero matrix isLet W2 be the subset of all symmetric matrices (all-zero matrix is

symmetric, linear combination of symmetric matrices is symmetric)

W2 is subspaceW2 is subspace

Example : Subset of all polynomials of even-power is subspace of the

space of n-th degree polynomialsp g p y

Example : Subset of all continuous real functions and subset of all

differentiable real functions are subspaces of the space of all real functions

SpanDefinition : Let V be a vector space. Vectors u1, u2, ….un in V span V if every

vector v in V is a linear combination of {u1, u2, ….un}, i.e.

v = a1u1 + a2u2 + …. + anunv a1u1 a2u2 …. anun

Remarks :

Find a span for X-Y plane.

1) Suppose {u1, u2, ….un} span V. Then, for any vector w, (assign zero

weight to w) the set {w, u1, u2, …., un} also spans V

Is it unique ?

2) Suppose {u1, u2, …., un} span V and suppose that uk is a linear

combination of some of the other u’s. Then,the u’s without uk also span V

Example : Verify that the vectors (1,1) and (-1,-1) don’t span R2 while the

vectors (1,1) and (1,0) do

Span Examples

Rspan100,010,001a)R space vector heConsider t 1)

3321

3

eee ),,(

Rspan100,010,001 a)

321

321

cebeaevcbaeee

)()(),,(

Rspan also001,011,111)

321

3321

wbawcbcwcbavwwwb

ittbt'2bbbtRitth th(

Rspan not does]9,5,1[],5,3,1[],3,2,1[,However)3

3213

b

uuuc

)u,u,u ofn combinatiolinear aswritten bet can'2bbbs.t.Rin any vector that show(

321

2313 b

More Span Examples for p p“Abstract” Subspaces

ofn combinatiolinear aasexpressedbecan polynomialEvery

n degree of spolynomial all : t)(PConsider )2 n

} t....., , t t,{1, )"0"about (expand pp yy

n2

matrices 4 by the spanned M spaceector Consider v)3 2,2

1000

,0100

,0010

,0001

,

Intersections & Unions of Subspacesp

u 0u

v

0su+tv

su tv

Vector Space

Theorem : The intersection of any number of subspaces of a vector V is

a subspace of V. Question : what about their union ? not a subspace! Gi t l (hi t id th d )Give a counter example, (hint : consider the x and y axes)

Proof: W U 0 W 0 U, 0Clearly V. of subspaces be W & Let U

subspaces)areWandU(sinceWbvauandUbvau e,Furthermor

W. vu, and U vu, W then U vandu suppose Now,

V of subspace a is W U W U bvau subspaces)are W andU(sinceW bvau and U bv au

The Null SpaceThe Null Space0Ax systemhomogenousaofset W solution The : Theorem

set.solutionthebeLet W:ProofA. of space null called R of subspace a is unknownsn in

ygn

0bAvaAubv)A(au0Avand0Au W vu, supposeW 000A

set.solution thebeLet W : Proof

notisbAxsethomogenous-nonofsetSolution:Remark W bv au

0bAvaAubv)A(au 0Avand 0 Au

it. tobelongt doesn' 0 since subspace anotisbAx set homogenousnonofset Solution :Remark

Connection to Linear EquationsAmxn matrix

rows : “m” n-dimensional vectors

columns : “n” m-dimensional vectors

Systems of Equations Ax=b can be thought of geometrically in as

1) The “A” matrix transforms vector “x” to vector “b”

2) The vector “b” is a linear combination of columns of “A” (lies in a vector space called the column space of A) So what is a vectorvector space called the column space of A). So, what is a vector space ?

b

xx

2

1

bvvv

n

n

x

221

bvvv nnxxx 2211

Row and Column Space of a Matrixo a d Co u Space o a at

Let A be m x n matrix. The rows (columns) of A may be viewed as

vectors in Rn (Rm); hence they span a subspace of Rn (Rm) called the

row space (column space) of A)R , ...... ,R ,(Rspan R(A)ncombinatioLinear m21

)C , ...... ,C ,span(C C(A)

A of (columns) rows of n21

001

:ofspacecolumn theDescribe :Example

3

Exercise : describe the row and null

f th 2 t i ll

100010 (i) 3

R spaces of these 2 matrices as well

line31

)(3-631-21

(ii)

AC

Connection to Linear Systems of Equations

Definition : the column space of A is the subspace of spanned by columns of Asubspace of spanned by columns of A

Th t f li ti A b hThe systems of linear equations Ax=b has1) No solution if and only if b is not in the

span of the column space of A2) Unique solution if b is in the span of ) q p

column space of A in a unique way3) Infinite solutions if b is in the span of the3) Infinite solutions if b is in the span of the

column space of A in infinite ways

Linear Dependence & Independencep p

linearly are Vin ......,,, vectorshesay that t We: Definition 21 vvv n

0......such that zero, allnot ,,......,, scalars ifdependent

2211

21

vavavaaaa

nn

n

n......,1,i0implies0

i.e. t,independenlinearly are vectors theOtherwise

ava i

n

ii

Relation to nullspace of a matrix

}1,tt,{1, vectorsofset theIs : Example

n......, 1,i 0implies0

3221

ttt

ava ii

ii

0)1()()()1( x:solution ?t independenlinearly

324

2321 tttxtxtx

t.independenlinearly isset hence ;0: (t)ofpowersfor tscoefficien equatingBy

1234 xxxx

Examples on Linear Independence

)5,9,4(),2,3,1(),0,1,1(Let)1 wvudependentlinearly0253Since wvu

020121tindependenlinearly )5,3,1(),7,5,2(),3,2,1(2)

zyxwvu

05730352

02

000

531

752

321

zyxzyx

zyxzyx

0002

05730573

zyx

zyx

0020

zyx

zzy

Linear dependence in R3 (geometric view)

a) Any 2 vectors in are linearly dependent v3Ra) Any 2 vectors in are linearly dependent

if and only if they lie on the same line through origin

b) Any 3 vectors in are linearly dependent

u

03R

R

b) Any 3 vectors in are linearly dependent

if and only if they lie on the same plane

s t1ki evectorsprecedingtheofncombinatiolinear a is vectors theof one Then, dependent.linearly are

vectorszero-nonmoreor 2Suppose : Theorem 21

m.., v, , vv

linearly are formechelon in matrix a of rows zero-non The :Theorem......

s.t 1k i.e. vectors,preceding theof

112211

kkk vcvcvcv

tindependen

Basis and Dimension

if)t(f

basis a is vectorsof ,.....,,set A : Definition 21V

uuuS n

tindependenlinearly is (ii) spans (i) ifspace)vector (for SVS

V

Exercise : give example of sets of vectors that satisfy only one of these 2 conditions

n m then elements, n"" has basisanother and elements m""has basis onesuch that space vector a be Let :Theorem

V

Exercise : give example of sets of vectors that satisfy only one of these 2 conditions

n.)dim( write weand VThe basis of a vector space is not uniquebut the number of vectors in the basisi i

iittbtb iiF2)

definitionby 0dimension has 0 space vector The 1) :Notes is unique

prove (2) by contradiction !

vectorsbasis theofn combinatiolinear a as way oneonly in written becan or every vectbasis,given aFor 2) by contradiction !

Basis and Dimension ExamplesBasis and Dimension Examples tindependenlinearly are ,......,, rsunit vecto The : 1) 21 n

n eeeR

bases.other Find basis.standard thecalled is This .)dim( ,for basis span & nnn nRRR

)di (

of basis a form n)j1 & mi1for whereelse

zerosandposition j)(i,in 1(hasmatrices The :)2

x

x

nm

ijnm

M

M

EmnM

shomogeneoutheofWspacesolutiontheofdimensionThe4)

1n dim basis a is ,.....,,,1:)()3

)dim( 2

x

n

n

nm

ttttP

mnM

unknownsofnumber theisn where”,variables“free ofnumber the is which r) -(n is A) of space null (i.e. 0 AX system

shomogeneoutheofW spacesolution theofdimension The 4)

pivots) of(number A ofrank theis “r” and,

Theorem

Th)di (htbL t VVdependent.linearly are Vin vectorsmoreor 1)(nAny (i)

Then,n. )dim(wherespace vector a be Let VV

n with ,......,,set t independenlinearly Any (ii)dependentlinearly are in vectors1)(nexample,For

21 uuuSR

n

n

elementsn with of ,......,,set spanningAny (iii) of basis a is elements

21 VvvvTV

n

Vspan not does Vin vectorslessor 1)-(n ofset Any (iv)V of basis a is

W

VV RV

3)di (Th

. of subspace a beLet W .3)dim(Let :Example 3

W(W)

W φ(W) W

i ith hlii1dib)point) (a 0dim a)

3. )dim( Then,

VW(W)W (W) W (W)

3did)origin through plane a is2dim c)

originthrough lineais1dim b)

VW (W) 3dim d)

:Example

nnnnnngularupper tria

nM n,n

)(Dim2)

)( Dim 1) 22

2

nnnnnmatricessymmetric

nngularupper tria

)(Dim3)

22)( Dim 2)

22

ndiagonal)

nmatricessymmetric

( Dim 4) 22

)( Dim 3)

Four Subspaces of A m x n

R) R(AC(A) R) C(AR(A)

mT

nT

or spaceColumn 2)or space Row )1

R) N(A RN(A)

)(( )

mT

n

space NullLeft 4) space Null 3)

p)

n - r(N(A)) r(C(A)) (R(A))

)(

dim 2) ""rank dim dim 1) :Facts

p)

m- r)) (N(AmnAA

T

T

dim )3

n) R (N(A)) (R(A))

nm

n

( dim dim dim 4)

Fundamental Theorem of Linear Algebra, Part I

m) R )) (N(A (C(A)) mT ( dim dim dim 5)

Row and Column Subspaces1) The columns of mxn matrix A are linearly independent when the rank r = n.

There are “n” pivots and no free variables. Only x = 0 is in the null space

2) The vectors v1, v2, ….., vn are a basis for Rn exactly when they are the

columns of an n x n invertible matrix. Hence, Rn has infinitely many

diff t b ( ll ith th di i “ ”)different bases (all with the same dimension “n”)

3) The pivot columns of A are a basis for its column space C(A). The pivotrows of A (or its echelon form) are a basis for R(A)( ) ( )

sameareandofspacesnullTherefore00......0 :Note 12

UAUxAxEEEAx k

same areandofspacesnull Therefore, UA

tindependenlinearlyofnumbermaximumthetoequalisAof rowst independenlinearly ofnumber maximum The 4)

)(rank )dim( )dim( Hence, A. of columnstindependenlinearly ofnumber maximum the toequal isA of

AC(A)R(A)

SpacesColumn and Row for the Bases Computing

Procedure 1(use to find a basis for the row space of A)

i Form the matrix M whose rows are the given vectorsi. Form the matrix M whose rows are the given vectors

ii. Row reduce M to echelon form

iii. The basis is given by non-zero rows of echelon matrixiii. The basis is given by non zero rows of echelon matrix

(or the corresponding rows in the original matrix)

Procedure 2 (use to find a basis for the column space)( p )

i. Form matrix M whose columns are the given vectors

ii. Row reduce M to echelon form

iii. For each column Ck in the echelon matrix without pivot, delete vector uk from the given list S

i Basis remaining ectors in Siv. Basis = remaining vectors in S

Example 1

)79641()813783()3,5,3,3,1(,)2,3,1,2,1( :by spanned of subspace a be Let

21

5

uu RW

ofsetthefromforbasisaFind)19,25,13,13,5(

)7,9,6,4,1(,)8,13,7,8,3(

5

43

uuuSWu

uu

5131151311

t)independen are which sother word(in ,.....,,ofset thefromfor basis a Find 521

u's

uuuSW

210003221051311

~13135

673148321311

:Sol

M

0000000000

1925

783291353

3)dim(for basis & delete 42153

W,u,uu W uu basis for column

space of M

Example 2Example 2

213121213121

211000121310213121

~9611673546552213121

BA

000000000000211000

1291186299810519611673 BA

for basis a Find ) R(A)a

Gaussian elimination applies elementary operations (linear combinations) on rows of A; Hence row subspace stays the same

for basis a formt independenlinearly are of rows zeroNon . space row same have and

R(A)BAB

Hence, row subspace stays the same

211000,121310,213121

Example 2 (Cont’d)

ti d dli lCCCi tithC lfor basis a Find ) C(A)b

for basis form

tindependenlinearly areC,C,Cpivots with Columns 421

C(A)

of formechelon in pivotscontain that of Columns1181163,65752,21321AA

)rank( Find ) Ac

rank3 )( dim (ii)3 )rank( pivots 3 (i)

R(A)A

rank 3 )dim( (iii))(( )

Col(A)( )

Example 3

1021ofspacecolumn andspace rowfor thebasisaandrank theFind 1)

5610333621021

10211021formechelon to Reduce :Solution

A

00001320~

26401320 ~

A

i( )fb ilii2 )rank( and itsfor basis a form )! itselfA of

rows 2first or ( of formechelon theof rows zero-non 2 The-AR(A)

A

)! formechelon its(not A of columns 2first by given isC(A)for basisaHence, 2.&1columnsin are Pivots-

32131matrix for this problem previousRepeat

Example 4

374324134132131

A

3213132131

8718337432

000001121032131

~336301121032131

~ U

0000011210

;for basis are of columns 2first ;for basis areAor of rows 2first ,2

C(A)AR(A)Ur

543 ,, are variablesfree325(A))dim(

xxxrnN

Find a basis for N(A)

Complete Solution to Ax=b

1

oSolution t Complete Find:3 Example

x

2

1

41

108223211

xxx

4

3 5131033xx

2440013211

U: formechelon in Put :1 Step

31:ablespivot variDetermine:2Step00000

xx

42

31

: variablesfree Determine

: ablespivot variDetermine :2Step

x,x

x,x

variablesfreeofin termsablespivot variWrite:3Step

xxxxx ;5.0

variablesfreeofin termsablespivot vari Write:3 Step

42143

x

0111

complete xxxx

x

5.0

01

001

:4 Step 423

2

pnull xxx

0104

pnull

Two basis vectors for null space of ATwo basis vectors for null space of A

Complete Solution to Ax = bnullparticularcomplete xxx

np

pnpcomplete bAxAxAxAxp

Procedure for computing the complete solution to Ax=b

variablesfree andpivot Determine 2)

formechelon in matrix augmentedPut )1 bA

ectorconstant v a plus)is(this variablesfree theof nscombinatiolinear as ablespivot vari theExpress 3)

nx) is this(

p)(

p

n

x

The 4 Fundamental SubspacesThe 4 Fundamental SubspacesFundamental Theorem of Linear Algebra (Part II)

Consider an m x n matrix A with rank r. The nullspace is the orthogonal complement (will be defined in Ch. 4) of the row space (in Rn). The left null space is the orthogonal complement of the column space (in Rm).

dim = rdim = r

AXp = b

X = Xp + XnRn Rm

column spacerow space

dim = m rdim = n r

AXn = 0left null space

null spacedim = m - rdim = n - r

Finding Basis for N(A) ffdi idb iFi dE l N(A)

2210031221

1132131221

(i)

:for ofdimension andbasis Find : Example

A

N(A)

0000022100~

5186311321 (i) A

3))(dim(,, variablesfree325,2

542 ANxxxrnr

752752322;22

1

54254321543

xxxxxxxxxxxx

20

20

01

5423

2

xxxxx

xnull

10

01

00

5

4

xx

ii) Write down the complete solution for Ax = b for b =

75233223;22 54254321543 xxxxxxxxxxx

03

07

05

12

2

1

54254321543

xx

00

02

12

00 542

4

3

2

xxxxxxcomplete

01005

4

x

321of )N(A space null left theofdimension and basis theFind T

02

022741642321

43

4321

xxxxxx

A

21212121

43243

00001200~

47633442AT

5.2222;5.0, variablesfree 2224))(dim(2

42432143

42

xxxxxxxxxxANr T

5005.2

012

422

1

xxxxx

x nullleft

15.0

00

4

3

xx

Linear Transformations

• Definition : A transformation T is linear iff :i) T(u+v)=T(u)+T(v)i) T(u+v) T(u)+T(v)

ii) T(cv)=cT(v)where c is a scalar and u,v are vectors,

• Definition : Let A be an m x n matrix. The function T(x)=y=Ax is called a matrix transformation fromT(x)=y=Ax is called a matrix transformation from

to mRnR

• Fact : Every matrix transformation on is linear (prove it!)

nR

Linear Transformation• How to determine its matrix representation ?

TvuTvuTvu ; ;;

143214321

nn2211

vvvvTuuuuTvuTvuTvu

;.....;;

• For V to be invertible, its “n” columns must be li l i d d t (i f b i )

1UVTTVU

linearly independent (i.e. form a basis)• Idea : if you know result of applying T(.) to

b i t k lt f l i T( )basis vectors, you know result of applying T(.) to any vector ! But which basis to choose ?T id t i i i V id tit t i !• To avoid matrix inversion, V = identity matrix !

Representation Theoremp

• Let T be a linear transformation from to . The m x n matrixnR mR

)()()( 21 nTTT eeeA

(where is the ith unit vector) has the property that T(x)=Ax and A is unique.

ie

Applying successive transformations corresponds to successive matrix multiplications

),....(),( 21 xTxT

corresponds to successive matrix multiplicationsxAA 12....

Examples• Rotation by angle counter clockwise

)cos()sin()sin()cos(

A

• Reflection about origin (mirror image) 01

Projection on x axis (drop a perpendicular)

10

A

• Projection on x-axis (drop a perpendicular)

01

ANot invertible ! Will be studied in detail in

00

AChapter 4