linear algebra in a nutshell - mcmaster universityqiao/courses/cs4xo3/slides/la... ·...

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Linear Algebra in A Nutshell

Gilbert Strang

Computational Science and EngineeringWellesley-Cambridge Press. 2007.

Outline

1 Matrix Singularity

2 Matrix Multiplication by Columns or RowsRank and nullspaceColumn space and solutions to linear equations

3 The Four Fundamental Subspaces

4 Dimension and Basis

Outline

Invertibility of an n-by-n matrix

A is invertibleThe columns are independentThe rows are independentThe determinant is not zeroAx = 0 has one solution x = 0

Ax = b has one solution A−1b

A has n (nonzero) pivotsA has full rank

A is not invertibleThe columns are dependentThe rows are dependentThe determinant is zeroAx = 0 has infinitely manysolutionsAx = b has no solution orinfinitely manyA has r < n pivotsA has rank r < n

Invertibility of an n-by-n matrix (cont.)

The reduced row echelon formis R = IThe column space is all of Rn

The row space is all of Rn

All eigenvalues are nonzeroATA is symmetric positivedefiniteA has n (positive) singularvalues

R has at least one zero row

The column space hasdimension r < nThe row space has dimensionr < nZero is an eigenvalue of AATA is only semidefinite

A has r < n nonzero (positive)singular values

Outline

Think of Ax a column at time

Instead of thinking of Ax inner products, think of Ax a linearcombination of columns of A:[

1 23 6

In particular,[1 23 6

]= first column

[1 23 6

]= last column

Think of Ax a column at time

Instead of thinking of Ax inner products, think of Ax a linearcombination of columns of A:[

1 23 6

]In particular,[

1 23 6

]= first column

[1 23 6

]= last column

In general

matrix-vector multiplication: y = Ax

column version

y = zeros(m,1);for j=1:n

y = y + x(j)*A(:,j);endfor

matrix-matrix multiplication: C = AB

column version (Fortran, step 1)C(:,j) = A*B(:,j)

In general

column version

In general

column version

In general

column version

Row version

vector-matrix multiplication: vT = uTA

row version

v = zeros(1,n);for i=1:m

v = v + u(i)*A(i,:);endfor

row version (C)C(i,:) = A(i,:)*B

Row version

row version

Row version

row version

Row version

row version

Rank and nullspace

Suppose A is an m-by-n matrix,Ax = 0 has at least one (trivial) solution, namely x = 0.There are other (nontrivial) solutions in case n > m.Even if m = n, there might be nonzero solutions to Ax = 0when A is not invertible.It is the number r of independent rows or columns thatcounts.

Rank and nullspace (cont.)

RankThe number r of independent rows or columns is the rank of A(r ≤ m and r ≤ n, that is, r ≤ min(m, n)).

Null spaceThe null space of A is the set of all solutions x to Ax = 0.

x in nullspace x1(column 1) + · · ·+ xn(column n) = 0

This nullspace N(A) contains only x = 0 when the columns of Aare independent. In that case A is of full column rank r = n.

Example. The nullspace of[

1 23 6

]is a line.

QuestionFind the line.

We often require that A is of full column rank. In that case, ATA,n-by-n, is invertible, and symmetric and positive definite.

Example. The nullspace of[

1 23 6

]is a line.

QuestionFind the line.

We often require that A is of full column rank. In that case, ATA,n-by-n, is invertible, and symmetric and positive definite.

Column (range) space

Column (range) spaceThe column (range) space contains all combinations of thecolumns.

Example. The column space of[

1 23 6

]is always through[

Column (range) space (cont.)

In other words, the column space C(A) contains all possibleproducts Ax , thus also called the range space R(A).

For an m-by-n matrix, the column space is inm-dimensional space.The word “space” indicates: Any combination of vectors inthe space stays in the space.The zero combination is allowed, so x = 0 is in everyspace.

Solution to linear equations

A solution to Ax = b calls for a linear combination of thecolumns that equals b. Thus, if b is in R(A), there is a solutionto Ax = b, otherwise, Ax = b has no solution.

How do we write down all solutions, when b ∈ R(A)?

Suppose xp is a particular solution to Ax = b. Any vector xn inthe nullspace solves Ax = 0.

The complete solution to Ax = b has the form:

x = (one xp) + (all xn).

Solution to linear equations (cont.)

Questions

Find the complete solution to[

1 23 6

Does the complete solution form a space?

Comments

Suppose A is a square invertible matrix, then the nullspaceonly contains xn = 0. The complete solutionx = A−1b + 0 = A−1b.

When Ax = b has infinitely many solutions, the shortest xalways lies in the “row space” of A. A particular solutioncan be found by the pseudo-inverse pinv(A).Suppose A is tall and thin (m > n). The columns are likelyto be independent. But if b is not in the column space,Ax = b has no solution. The least squares methodminimizes ‖Ax − b‖2

2 by solving ATAx̂ = ATb.

Comments

Suppose A is a square invertible matrix, then the nullspaceonly contains xn = 0. The complete solutionx = A−1b + 0 = A−1b.When Ax = b has infinitely many solutions, the shortest xalways lies in the “row space” of A. A particular solutioncan be found by the pseudo-inverse pinv(A).

Suppose A is tall and thin (m > n). The columns are likelyto be independent. But if b is not in the column space,Ax = b has no solution. The least squares methodminimizes ‖Ax − b‖2

Comments

Suppose A is a square invertible matrix, then the nullspaceonly contains xn = 0. The complete solutionx = A−1b + 0 = A−1b.When Ax = b has infinitely many solutions, the shortest xalways lies in the “row space” of A. A particular solutioncan be found by the pseudo-inverse pinv(A).Suppose A is tall and thin (m > n). The columns are likelyto be independent. But if b is not in the column space,Ax = b has no solution. The least squares methodminimizes ‖Ax − b‖2

Outline

Four spaces of A

The column space R(A) of A is a subspace of Rm. Thenullspace N(A) of A is a subspace of Rn. In addition, weconsider N(AT), a subspace of Rm and R(AT), a subspace ofRn.

Four spaces of A: R(A), N(A), N(AT), R(AT)

QuestionLet

[1 23 6

Draw R(A) and N(AT) in the same figure, and draw N(A) andR(AT) in the another figure.

Four spaces of A

The column space R(A) of A is a subspace of Rm. Thenullspace N(A) of A is a subspace of Rn. In addition, weconsider N(AT), a subspace of Rm and R(AT), a subspace ofRn.

Four spaces of A: R(A), N(A), N(AT), R(AT)

QuestionLet

[1 23 6

Draw R(A) and N(AT) in the same figure, and draw N(A) andR(AT) in the another figure.

Four spaces of A (cont.)

Four subspaces

R(A) and N(AT) are perpendicular (in Rm).N(A) and R(AT) are perpendicular (in Rn).

Each subspace contains either infinitely many vectors or onlythe zero vector. If u is in a space, so are 10u and −100u (andmost certainly 0u). We measure the dimension of a space notby the number of vector, but by the number of independentvectors. In the above example, a line has one independentvector but not two.

Four spaces of A (cont.)

Four subspaces

R(A) and N(AT) are perpendicular (in Rm).N(A) and R(AT) are perpendicular (in Rn).

Each subspace contains either infinitely many vectors or onlythe zero vector. If u is in a space, so are 10u and −100u (andmost certainly 0u). We measure the dimension of a space notby the number of vector, but by the number of independentvectors. In the above example, a line has one independentvector but not two.

Outline

A basis for a space

A full set of independent vectors is a basis for a space.

Basis1 The basis vectors are linearly independent.2 Every vector in the space is a unique combination of those

basis vectors.

Some particular bases for Rn:

standard basis = columns of the identity matrixgeneral basis = columns of any invertible matrixorthogonal basis = columns of any orthogonal matrix

The dimension of a space is the number of vectors in a basisfor the space.

A basis for a space

basis vectors.

A basis for a space

basis vectors.

linear algebra in a nutshell - mcmaster universityqiao/courses/cs4xo3/slides/la... ·...

Documents